lessons/4-ComputerVision/08-TransferLearning/Dropout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Effect of Dropout\n",
    "\n",
    "Let's see for ourselves how dropout actually affects training. We will use MNIST dataset and a simple convolutional network to do that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
    "\n",
    "x_train = x_train.astype(\"float32\") / 255\n",
    "x_test = x_test.astype(\"float32\") / 255\n",
    "\n",
    "x_train = np.expand_dims(x_train, -1)\n",
    "x_test = np.expand_dims(x_test, -1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define `train` function that will take care of all training process, including:\n",
    "* Defining the neural network architecture with a given dropout rate `d`\n",
    "* Specifying suitable training parameters (optimizer and loss function)\n",
    "* Doing the training and collecting the history\n",
    "\n",
    "We will then run this function for a bunch of different dropout values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training with dropout = 0\n",
      "Epoch 1/5\n",
      "938/938 [==============================] - 26s 27ms/step - loss: 0.1949 - acc: 0.9435 - val_loss: 0.0596 - val_acc: 0.9802\n",
      "Epoch 2/5\n",
      "938/938 [==============================] - 27s 29ms/step - loss: 0.0592 - acc: 0.9816 - val_loss: 0.0433 - val_acc: 0.9857\n",
      "Epoch 3/5\n",
      "938/938 [==============================] - 26s 28ms/step - loss: 0.0438 - acc: 0.9867 - val_loss: 0.0472 - val_acc: 0.9849\n",
      "Epoch 4/5\n",
      "938/938 [==============================] - 27s 28ms/step - loss: 0.0355 - acc: 0.9890 - val_loss: 0.0353 - val_acc: 0.9882\n",
      "Epoch 5/5\n",
      "938/938 [==============================] - 26s 28ms/step - loss: 0.0294 - acc: 0.9910 - val_loss: 0.0305 - val_acc: 0.9894\n",
      "Training with dropout = 0.2\n",
      "Epoch 1/5\n",
      "938/938 [==============================] - 29s 31ms/step - loss: 0.2097 - acc: 0.9377 - val_loss: 0.0655 - val_acc: 0.9781\n",
      "Epoch 2/5\n",
      "938/938 [==============================] - 31s 33ms/step - loss: 0.0676 - acc: 0.9792 - val_loss: 0.0409 - val_acc: 0.9852\n",
      "Epoch 3/5\n",
      "938/938 [==============================] - 28s 30ms/step - loss: 0.0514 - acc: 0.9837 - val_loss: 0.0384 - val_acc: 0.9871\n",
      "Epoch 4/5\n",
      "938/938 [==============================] - 28s 29ms/step - loss: 0.0424 - acc: 0.9871 - val_loss: 0.0343 - val_acc: 0.9889\n",
      "Epoch 5/5\n",
      "938/938 [==============================] - 30s 32ms/step - loss: 0.0356 - acc: 0.9893 - val_loss: 0.0343 - val_acc: 0.9885\n",
      "Training with dropout = 0.5\n",
      "Epoch 1/5\n",
      "938/938 [==============================] - 30s 31ms/step - loss: 0.2586 - acc: 0.9212 - val_loss: 0.0666 - val_acc: 0.9797\n",
      "Epoch 2/5\n",
      "938/938 [==============================] - 28s 30ms/step - loss: 0.0860 - acc: 0.9734 - val_loss: 0.0441 - val_acc: 0.9860\n",
      "Epoch 3/5\n",
      "938/938 [==============================] - 29s 31ms/step - loss: 0.0674 - acc: 0.9792 - val_loss: 0.0414 - val_acc: 0.9868\n",
      "Epoch 4/5\n",
      "938/938 [==============================] - 30s 32ms/step - loss: 0.0564 - acc: 0.9822 - val_loss: 0.0326 - val_acc: 0.9886\n",
      "Epoch 5/5\n",
      "938/938 [==============================] - 29s 31ms/step - loss: 0.0511 - acc: 0.9843 - val_loss: 0.0298 - val_acc: 0.9899\n",
      "Training with dropout = 0.8\n",
      "Epoch 1/5\n",
      "938/938 [==============================] - 31s 32ms/step - loss: 0.3832 - acc: 0.8766 - val_loss: 0.0849 - val_acc: 0.9732\n",
      "Epoch 2/5\n",
      "938/938 [==============================] - 29s 31ms/step - loss: 0.1563 - acc: 0.9521 - val_loss: 0.0686 - val_acc: 0.9797\n",
      "Epoch 3/5\n",
      "938/938 [==============================] - 32s 34ms/step - loss: 0.1253 - acc: 0.9616 - val_loss: 0.0490 - val_acc: 0.9854\n",
      "Epoch 4/5\n",
      "938/938 [==============================] - 33s 35ms/step - loss: 0.1105 - acc: 0.9658 - val_loss: 0.0395 - val_acc: 0.9872\n",
      "Epoch 5/5\n",
      "938/938 [==============================] - 34s 36ms/step - loss: 0.1022 - acc: 0.9680 - val_loss: 0.0363 - val_acc: 0.9878\n"
     ]
    }
   ],
   "source": [
    "def train(d):\n",
    "    print(f\"Training with dropout = {d}\")\n",
    "    model = keras.Sequential([\n",
    "        keras.layers.Conv2D(32, kernel_size=(3, 3), activation=\"relu\", input_shape=(28,28,1)),\n",
    "        keras.layers.MaxPooling2D(pool_size=(2, 2)),\n",
    "        keras.layers.Conv2D(64, kernel_size=(3, 3), activation=\"relu\"),\n",
    "        keras.layers.MaxPooling2D(pool_size=(2, 2)),\n",
    "        keras.layers.Flatten(),\n",
    "        keras.layers.Dropout(d),\n",
    "        keras.layers.Dense(10, activation=\"softmax\")\n",
    "    ])\n",
    "    model.compile(loss='sparse_categorical_crossentropy',optimizer='adam',metrics=['acc'])\n",
    "    hist = model.fit(x_train,y_train,validation_data=(x_test,y_test),epochs=5,batch_size=64)\n",
    "    return hist\n",
    "\n",
    "res = { d : train(d) for d in [0,0.2,0.5,0.8] }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's plot validation accuracy graphs for different dropout values to see how fast the training goes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x235bc70f0d0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAD4CAYAAAAHHSreAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABMqUlEQVR4nO3dd3gU1frA8e+b3gtJgDQIXQIIYkCk2BDFckWxoaBIEVBQ7GK/tqteu1cUUURQKerVn+WqgNio0kVDr6mQRnrd3fP7Y4awCYFsIMlukvN5nn2yOzNn5p1JMu+eM2fOiFIKTdM0TbPn5uwANE3TNNejk4OmaZp2HJ0cNE3TtOPo5KBpmqYdRycHTdM07Tgezg6gPoSHh6u4uDhnh6FpmtakbNy4MUspFVHTvGaRHOLi4tiwYYOzw9A0TWtSROTgiebpZiVN0zTtODo5aJqmacfRyUHTNE07TrO45lCTiooKUlJSKC0tdXYop83Hx4eYmBg8PT2dHYqmaS2EQ8lBRIYDbwLuwAdKqRerzQ8FPgQ6AaXAeKXU3+a86cDtgADvK6XeMKe3AhYDccAB4Aal1BFz3iPABMAK3K2UWlLXHUtJSSEwMJC4uDhEpK7FXYZSiuzsbFJSUujQoYOzw9E0rYWotVlJRNyBmcBlQDxwk4jEV1vsUWCLUupM4FaMRIKI9MRIDP2B3sCVItLFLDMDWK6U6gIsNz9jrnsU0AMYDrxjxlAnpaWlhIWFNenEACAihIWFNYsakKZpTYcj1xz6A3uUUvuUUuXAImBEtWXiMU7wKKV2AHEi0gboDqxVShUrpSzAb8A1ZpkRwDzz/Tzgarvpi5RSZUqp/cAeM4Y6a+qJ4ajmsh+apjUdjiSHaCDZ7nOKOc3en8BIABHpD7QHYoC/gfNEJExE/IDLgVizTBulVDqA+bN1HbaHiEwSkQ0isiEzM9OB3dA0TWseiiuK2Xh4I/MT5/PjgR8bZBuOXHOo6Wtr9YdAvAi8KSJbgL+AzYBFKbVdRF4ClgGFGEnEUg/bQyk1G5gNkJCQ4LIPpfjxxx+ZPn06VquViRMnMmPGDGeHpGlaE1JcUczOIzvZlr2NxKxEErMT2Z+3H2WeFi/rcBnD44bX+3YdSQ4pHPu2D0aNIM1+AaVUPjAOQIw2kP3mC6XUHGCOOe9f5voADotIpFIqXUQigQxHt9dUWK1Wpk6dyrJly4iJiaFfv35cddVVxMdXv2SjaZoGpZZSdh7ZSWJWopEMshPZl7cPm7IBEO4bTo+wHgyPG06P8B7Eh8UT7hveILE4khzWA11EpAOQinGx+Gb7BUQkBCg2r0lMBH43EwYi0loplSEi7TCans41i30DjMWodYwFvrabvkBEXgOigC7AulPeQydat24dnTt3pmPHjgCMGjWKr7/+WicHTdMot5az68iuytrAtuxt7Mndg1VZAWjl04oeYT0Y2m4oPcJ60CO8B639Wtey1vpTa3JQSllEZBqwBKMr64dKqUQRmWLOn4Vx4Xm+iFiBbRjdUI/6r4iEARXA1KPdVTGSwmciMgFIAq4315coIp+Z67GYZayns5NPf5vItrT801nFceKjgnjqHz1OukxqaiqxsccqQTExMfzxxx/1Goemaa6vwlrB7tzdJGYnVtYKdufuxmIzWtlDvEPoEdaD82LOq0wEbfzaOLUzikP3OSilvge+rzZtlt37NRjf8GsqO+QE07OBoSeY9zzwvCOxubKans+tex5pWvNWYatgX+6+ykSQmJ3IriO7qLBVABDoFUiPsB6MjR9b2TQU5R/lcueGZnuHtL3avuE3lJiYGJKTj3W8SklJISoqyimxaJpW/yw2C/vz9lepEew8spMyaxkAAZ4BxIfFM6b7GOLD4+kR1oOYgJjTSgRWm2J/VhHb0vPZnp5PXJgfN/ZrV1+7VKlFJAdn6devH7t372b//v1ER0ezaNEiFixY4OywNE07BVablYP5B41EYCaDHTk7KLUaN6j6efjRPaw7N3a7sbJpKDYwFjc59SHsisos7DhUwLb0fLalGclgx6F8SiuMC9QebsL1CbHc2K9edrEKnRwakIeHB2+//TaXXnopVquV8ePH06OHc2oxmqY5zqZsHMw/WNljKDErke052ymxlADg6+FL91bdua7rdcSHxdMjvAdxQXGnnAiUUmQUlLEtLb9KItifXcTR1ukgHw/io4K4uX974qOC6B4ZSOfWAXh71HkACYfo5NDALr/8ci6//HJnh6Fp2gkopUguSD6WCMyeQ0UVRQB4u3tzRqszuKbzNUYiCOtBh+AOuLud2knZYrWxL6vouESQXVReuUxsK1/iI4MY0Sea+Kgg4qOCiAr2adTrEjo5aJrWYiilSC1MrUwAR38WlBcA4OXmRbdW3biy45X0CDMuFncK6YSH26mdKgtKK4xmobT8ymSw83AB5RajWcjL3Y1ubQMZ2r018ZFBxEcFc0ZkIEE+zh+BWScHTdOaJaUUh4oOVUkEidmJ5JXlAeDh5kHX0K7GDWXmNYJOIZ3wdKv7iVkpRVpeKdvtagPb0vNJyimuXCbUz5MeUcGMPddoFoqPDKZjhD+e7q75WB2dHDRNc76SI7DvV+NlKQOfYPAOMn6e7GU27SilyCjOqNIstC17GzmlOQB4iAddQrtwcbuLK68RdAnpgpe7V51DLbfY2JNRyPb0qokgr6SicpkO4f70ig7mhoSYykTQJsjb5bqrnoxODpqmNT6bFdK2wJ6fjFfqBlA28A4G32AozYPSfGoYVg2ALHc3Er28SPQLJNHHh20ebmS5Gcu6A53cAznPJ5weEd3pEdSBriGd8PYLt0ssITgy7mhecUVll9GjiWB3RgEVVmNb3h5unBEZxOW9Is0kEEi3tkEEeDf9U2vT3wNN05qGgkOw92cjGez92agtIBDdF857EDoNheizwd08LdlsUF5Adt5BtmX8SWLOdhJz97CtKIWMCuMagRvQ0c2PgXgTb4Ue5RV0Ky7EtzQVyrYbCedkvALAJxjlE0y5RwB5Nj+yrT6kl3mTVORJaqkX+fiTr/wQn2D6RbTmqoS2xEVH0Tk2krjWIXi4aLPQ6dLJQdO0hmEph+S1sGe58Tr8lzE9oA10vQw6D4WOF4J/WGWRvLI8Eg8lVmkeSi9KB0AQ4oLj6B97fuXF4jNanYGfp1/N27fZoLzQrIUce1UU55KZmUFOdib5uVmUFuRgzc/Fz1pEkGQQJMUkSAkXUISbp11ysQGHzdef5jRPf7Mm4kATmH2t5WizmUfdm7Uai04ODay2Ibs//fRTXnrpJQACAgJ499136d27tzNC1bTTl7P/WM1g/+/GydnNE9oNgIv/CZ0vhjY9wWx7t9qs/JWxhdVpq1mVuoq/sv6qHIq6fVB7+rTuw+iw0cSHxdO9VXcCvAIcj8XNjRyrD9uzytmWVsH2dAvb0q3syQCLLRjogp+XO2e0DSS+m3FdID4qiNZtAvH1cgelakgu+dU+51b9XJgBWbuPfa5tWDgP3zokFbvEcjQZeXifym/JITo5NCBHhuzu0KEDv/32G6Ghofzwww9MmjRJD86nNR3lRXBgpXntYDnk7DWmh7SHM280kkGHIeAdWFnkcNFhIxmkrWJN2hryy/NxEzd6hvdkSu8pJLRJ4IywMwjyCnI4DJtNkZRTXOW+gW3p+aTnHXu8btsgH7pHHu02aiSC9q38cHM7wUViESNu70AIjqn7sVHKOD6leVBWPanUkFhK86A4yziGRz/bann8jYcP9LwOrp5Z9/hqoZNDA3JkyO6BAwdWvh8wYAApKSnHrUfTXIZSkLHdrB0sh4OrwVoOnn4QNwTOmWwkhFYdK2sH5dZyNqWvZVXqKlalrWL3kd0AtPZtzUXtLmJQ9CDOjTyXYO9gh0IorbCys9qQEtvT8ykqN76lu7sJnSMCGNAxjO6RgcRHBtM9MpCwgIb7ll0jEfAOMF7HP8yydkpBRUkNSaVaYmndvb4jB1pKcvhhBhz6q37X2bYXXPbiSRep65Ddc+bM4bLLLqu3EDWtXhztZrrnJ9jzMxSYz95qHW8kg05Dod254OlTWSQpP4mVqStZlbaK9YfWU2IpwdPNk75t+nLf2fcxKHoQXUK61Nq1M6vw+CEl9mYWYjM7MQV4exAfGcT1CbGViaBLmwB8PBtmSIlGJQJefsYrKLLRN98ykoOT1GXI7l9++YU5c+awcuXKhg5L007uRN1MfYKNC8idhxoJIfjYt+HiimLWJf9qJITUVaQUGjXgdoHtuLrz1QyKGkS/tv1OfPEY4//l112ZrNufU5kIMgrKKudHh/jSPTKIy3pFEm8mgphQ3xM3C2mnpWUkh1q+4TcUR4fs3rp1KxMnTuSHH34gLCzsuPma1uDq2M1UKcWunJ2sSlvFqtRVbMrYhMVmwdfDl3PansPYHmMZFDWI2KDYk2/XXNcvOzN4bdku/k7Nx9Nd6Nw6kCFdIsx7B4xB5kL8XLdnT3PUMpKDkzgyZHdSUhIjR47k448/pmvXrk6KVGtxTqGbaW5pLmuT1rIydSWr01aTWZIJQNfQrtwSfwuDowbTp3Ufh+86Vkqxak82ry7byeakXGJb+fLK9b35R+/IBhtpVHOcTg4N6ERDds+aZTxEb8qUKTzzzDNkZ2dz5513VpbZsGGDM8PWmqt66mYa5BXEwKiBDIoexMCogaf0XON1+3N4delO/tifQ1SwDy+M7MV1Z8e47DhDLZHU1C7e1CQkJKjqJ9Tt27fTvXvDXMV3hua2P1ojOFk3084X17mb6eCowQyKHkSPsB6nPFz15qQjvLZsFyt2ZxER6M20Czszqn+srik4iYhsVEol1DRP1xw0rbmw72a65ydIWtPo3UxP5O/UPF5ftovlOzJo5e/FY5d3Z8yA9sbNZtpJKaWwFRVhyczEmpWFJSsLS6b5MysLn+7daXXLmHrfrkPJQUSGA29ijGn1gVLqxWrzQ4EPgU5AKTBeKfW3Oe9eYCLGCFp/AeOUUqUishjoZq4iBMhVSvURkThgO7DTnLdWKTXllPdQ05ozJ3YzdcSuwwW8vmwXP/x9iCAfDx68tBtjB8Y1i4HpTpetrOzYyb7KCT8TS1YW1qOfs7NRpaXHr8DDA4+wMNwD63DXeB3U+hsSEXdgJjAMSAHWi8g3Sqltdos9CmxRSl0jImeYyw8VkWjgbiBeKVUiIp8Bo4CPlFI32m3jVSDPbn17lVJ9TnPfNK35sVkhbbN5Ifn0u5mO6DSCwdGDa+1mWlf7Mgt5c/luvvkzDX8vD+4e2oUJgzsQ7Ov8h9g0JGWxYMnJwZqdfdwJ31rtG7+toKDGdbi3aoVHWBgeEeH4tu+LR3gEHuHheESEGz/Dw3EPD8c9OBhxa7hrNI6k7/7AHqXUPgARWQSMAOyTQzzwAoBSaoeIxIlIG7tt+IpIBeAHpNmvXIyvJzcAF53Ojmhas1VwyEgGe5fXSzfTW3vcyqCoQbQLalfvoSbnFPPW8t18uTkVL3c3ppzfiUlDOhLq33S7oSqlsOXlHfuGn5Vd48nekpWFNScHariO6xYQUHli9+7WDf9Bg6qc8N3Dw40k0CoU8XSNBOpIcogGku0+pwDnVFvmT2AksFJE+gPtgRil1EYReQVIAkqApUqppdXKDgEOK6V2203rICKbgXzgcaXUCof3SNOaugboZjooahBntT7rlB5u44j0vBLe/nkPi9cn4+Ym3DYwjinndyIisJGHrKgDW3Fx1Sad7KyaT/hZWaiKiuPKi5eXcWKPCMczJgbf3r0rT/juZiLwiIjAIywMN19fJ+zh6XEkOdTU8Fg9Nb4IvCkiWzCuK2wGLOa1iBFAByAX+FxExiilPrErexOw0O5zOtBOKZUtImcD/yciPZRS+VWCEpkETAJo167+vwFpWqM62s10z3I4sMKp3UzrIqOglHd/3cunfyShlOKm/u2YemFn2gb71F64Aajyciw5OeYJPrPy5G5827dr4snMwlZcfPwK3NxwD2tV2ZTj3bmz+Y0/7Ni3e/PbvltgYJN6sltdOZIcUgD72xxjqNY0ZJ64x0FlM9F+83UpsF8plWnO+xIYCHxifvbAqHGcbbeuMqDMfL9RRPYCXYEqfVWVUrOB2WB0ZXVob52gtiG7f/31V0aMGEGHDh0AGDlyJE8++aQzQtUaU5Vupj9Bzj5j+imOZnpH7ztOu5tpXeQUlfPeb3uZt+YAFVbFdX1juGtoZ2JC6++6xVHKZsOam1t5wq/SYye7ahOPNTe3xnW4BQdXNuv49uxV9Rt+mF3zTmgo4q57UIFjyWE90EVEOgCpGBeUb7ZfQERCgGKlVDlGz6TflVL5IpIEDBARP4xmpaFUPclfDOxQSqXYrSsCyFFKWUWkI9AF2HeqO+hMjgzZDTBkyBC+++47J0WpNYpau5lOcbibaYRvRL12M62LvJIKPlixjw9X7qe4wsrVfaKZPrQLceH+p7VeZbVSmphI0erVlCclG008dr11sB7/XATx9a084XvFdcCvX79jzTlHv+GHGd/43bya7jUPZ6k1OSilLCIyDViC0ZX1Q6VUoohMMefPAroD80XEinGheoI57w8R+QLYBFgwmptm261+FFWblADOA54REQtgBaYopXJOYx+dxpEhu7VmrDTv2HhFp9HN1MPNg7Nbn13v3UzrorDMwtyV+3l/xT7ySy1c0SuSey7uQpc2gbUXPoGKjAyKVq2maMUKilavNr71i+DRpo3ZWycC7/jux0721XrsuPmfXkLSTs6hzsZKqe+B76tNm2X3fg3GN/yayj4FPHWCebfVMO2/wH8dictRL617iR05O+pzlZzR6gwe7v/wSZdxdMjuNWvW0Lt3b6KionjllVfo0aNHvcaqNRKlIHsP7FoCu340agc2yyl1M40NjG2wbqZ1UVJuZf6aA8z6bS9HiisYFt+Gey/uSnyU4w/iOUqVl1O8eQtFK1dQuGIlZTuM/0n38HACLrgA/8GD8R80EI/Q0PreDe0U6DtRGpAjQ3b37duXgwcPEhAQwPfff8/VV1/N7t27jyunuShLORxcBbuXGgnh6LWD1vEw8C7ocinE9HOJbqZ1UVphZeG6JGb+speswjLO7xrBfcO60js2pE7rKU9OpmjlSgpXrKR47VrjIrCHB359+xJx/30EDB6Md7duDdpfXzs1LSI51PYNv6E4MmR3UNCxb2CXX345d955J1lZWYSHhzdanFodFWaYyWAJ7P0FygvA3Rs6nAcD7oSul0LIsZO7s7uZ1kW5xcbnG5N5++c9pOeVcm7HMGaN6UtCXCuHytuKiylat46ilasoWrGC8oMHAfCMiSFoxFUEDBmCX/9zcA/QTUKurkUkB2dxZMjuQ4cO0aZNG0SEdevWYbPZ9DMdXI1SkP6nkQx2L4HUjcb0wCjodS10HU5xTAKp5bmkFaaRdmgVaXs+J7UwleSCZHbm7KzSzfRoV9OG7mZaFxarjS83p/LW8t2kHCnh7PahvHp9bwZ2PvmXFKUUZbt3G8lg5QqK129AVVQgPj74ndOf0DFjCBgyGM/27Zt1t8/mSCeHBuTIkN1ffPEF7777Lh4eHvj6+rJo0SL9T+QKyouMMYt2/Qi7l1FUeIhUD0/S2p5B6tnXkh4YQZoqJ7UojbTN/yJ3bW6V4t7u3kT6RxIdEM0dve9gYPRAeob1bJRupnVhtSm+25rGGz/tZn9WEb2ig3nu6p6c3zXihH+H1rw8itasoXDlSopWrMRy+DAA3l06VyYD37PPxs3bdW+A02qnh+xuIprb/riawvJCUtPWk7ZnCWlp60jNO0C6u5Dq6UWalzd5VO1K6ePuQ1RAFJEBkUT7RxMVEEV0gPEzKiCKMJ8wl07yNptiSeIhXv9pF7sOF3JG20DuG9aVYfFtjov7aDfTwhUrKFq5ipI//wSbDbegIPzPPZeAIYPxHzwYz7ZtnbQ32qnSQ3ZrLV5BeQFphWmkFqZW/kwvTCPtyG5Siw6Rr6oOj+AbHESUbwRRIZ04MzCm8qR/NBG08mnl0if/E1FK8fOODF5duott6fl0ivDn7ZvP4vKekVWexXyibqY+vXoRPmUy/oOH4HtmL8RDn0KaK/2b1ZqF/PL8Kif/ygRQlE5qYSoF5VVHwPTFjeiKCqIqyjnTaiM6MIaoyH5EdxpGVHR/Qr1Dm+TJ/0SUUqzck8WrS3exJTmX9mF+vHZDb0b0icbdTVDl5RTpbqaaHZ0cNJenlKo8+VcmgKKj3/7TSStMo6Ci2snfw5fogGiiA6LoE9Ce6KIcojL2En14O1EVFYT4tEK6Xmr0LOp4IfjUvd9+U/HHvmxeXbqLdQdyiA7x5aVrezGybwwqLZX8xYtq7mZ6330EDNHdTFsynRw0pzt68rdv8qlMBEVGAiisKKxSxs/Dj+jAaKL9ozm7zdmVzT5RAVFEe4cRnLYV2b0Uti6BvCSjUNsz4Zzp0HU4RJ0FzfyktynpCK8t3cXKPVm0DvTmueGduMKaRtmSj0l63K6baXS00c108GD8zhmgu5lqgE4OWiNQSpFXlkdqUdUmH/umn6KKoipl/D39zW/+0fRv27+y58/RC79BXkFVm33y04yupqs/NHoZVRQb4xZ1vADOux+6XAJBVe8xaa7+Ts3jtWW7+Hn7YXpbsnnHL5PuuxMpW7SRQ9W6mfoPHoRXXFyzakLT6odODtppU0qRW5Z73Ek/rehYIiixlFQpE+AZQHRANLGBsQyIHFDl5B8VEHX8yb86m82432DXj0ZSOLTVmB7SDvqMNmoHcYOrjFvU3O08VMA732wme8VKzs3ezV05u/HJzQZAmd1M/QcPwi8hQXcz1Wqlk0MDq23I7ry8PMaMGUNSUhIWi4UHHniAcePGOSnak1NKsTptNbuO7KrS9JNWlHbcyT/QK5DogGjaBbZjQOSAY00+ZgII8jqFNv7SfGMgu11LYM8yKMoEcYPYAXDx08b1g4gzKkc1bQmU1crelRtYtfA7ArZuYNKRg7gphQQGEjBwoO5mqp0ynRwakCNDds+cOZP4+Hi+/fZbMjMz6datG6NHj8bLxYYYVkrx+qbXmfv3XACCvIKIDogmLjiOgdEDifI/dvKPDIg8tZN/TbL3mrWDH+HganMguxDoMswYt6jzUPBzbGiH5uJoN9PM5b9SuHo1PsUFJCDktetE0LW3E3bhBbqbqXba9F9PA3JkyG4RoaCgAKUUhYWFtGrVCg8X+6dWSvHyhpf5eNvH3NjtRqb3nU6g16kP1XxSlnJjNNOjI5vm7DWmR3SHc6cZtYOY/pUD2bUEJxrN9Ih3IJvbdsN/8GAuHzeCHjG6dqDVnxbxH3boX/+ibHv9Dtnt3f0M2j766EmXcWTI7mnTpnHVVVcRFRVFQUEBixcvxs2FetEopXhh3Qss3LGQ0d1H83C/h+v/4mVhptFMtOtH47kHlQPZDYEBdxgXk0Pb1+82XVyNo5m6e3C4fTeW9LyCjW26MeDic5g6tCttglrOdRWt8bSI5OAsjgzZvWTJEvr06cPPP//M3r17GTZsGEOGDKkyWquz2JSN59Y+x+e7Pmds/FiujZvCxHkbEBFiQn3Nl1/l+2BfT8cSh1LGBeRd5jDXqRsBBYGR0HOkcTG54/ng1XK6VJ5wNNPoaLwvv4JfgjrxZk4QBW7e3JAQw9yLuhAd0vQeWq81HS0iOdT2Db+hODJk99y5c5kxYwYiQufOnenQoQM7duygf//+jR1uFTZl4+k1T/Pl7i+Z2Gsi54aOYeS7q7EpiAz2Ye2+bArLLFXKBHh71Jg0okP8iPG3EXJ4tXHvwa6l5lPRBKL7woWPGs1Fbc9sMReTHRnNVCWcw9yDVuauPkDJEStX9zUeydk+rOUkTc15WkRycBZHhuxu164dy5cvZ8iQIRw+fJidO3dWXqNwFqvNypOrn+Sbvd8wpfcUomwjGPPBOmJCffnwtn7EhfsbN66VWEg+UkzKkRJSKn8a7//Yl0NweToXum3mIrfNdHXbhkgFRfiy3S+B5Ha3UdTuIlq1iTGSSJAfoUBzTg2Vo5maA9hVH830aDfTIuXG3FUHeH/xPgpKLVx5ZiT3XNyVzq0DnLwHWkuik0MDcmTI7ieeeILbbruNXr16oZTipZdecuqDfiw2C4+tfIzv93/P1D5TKc28iHt/+pMBHVsxa8zZhPgZvahEhGA/T4L9gukZbT7g3mqBlPWwa7nxMJyMbQAUBbRnd+gNbPI5h9WWrhzMtZCaVEz+rgwgo3Lbfl7uVWod0SFVayCt/L2a3M1apTt3UvDTT1VHMw0MxP9oN9NBg/CMjASguNzC7DUHmfXbXnKLK7gkvg33DutK90jnNzFqLY8esruJaIz9qbBVMOP3GSw9uJRpfe5mx45+fLU5lWv7xvDCyF54edRwobw4x7z34EfY8xOUHAE3D2g/0Lh20OVSCO9c4/bySipIraHWkZprvM8rqTZSqqf7saaqKk1Xxs8wF0seRxYu5NCzz4FS+PTsad5zcPxopqUVVj79I4l3f91DVmE5F3aL4L5h3egVE+zE6LWWQA/ZrdWqwlrBg78/yPKk5dx55j0sXxPPugOpPHBJV6Ze2PnYSVcpyNx57M7k5D9AWcEvHLpeBl0vgU4XgU/tJ7ZgX0+CfT1P+LD6/NKjyeNYAkk9UkJKbjGbk3PJLa6aPHw83YgJ9TNrHFVrHTGhfoQHNE7yUEqR+drrZL//PgEXXEDkv57Ho9Xx92KUW2ws3pDMzJ/3cCi/lEGdw3hvWDfObq9HPtWcz6HkICLDgTcBd+ADpdSL1eaHAh8CnYBSYLxS6m9z3r3AREABfwHjlFKlIvJP4HYg01zNo0qp780yjwATACtwt1JqyenspHZy5dZy7v/1fn5N+ZWJ8fey+KcOpOXl8tZNZ3FVb7sL6Dn7YMEoyNppfG7bC4bcZ9QOovtCPT/lLMjHk6BIzxM2qxSUVhi1jJyqNY6UIyVsTcnlSLXk4e3hVq3GUfV9RID3aScPVV5O+hNPkPf1N4TceCNtn3j8uJvRLFYbX25K5a2fjUdy9osL5fUb+3BuJ/14WM111JocRMQdmAkMA1KA9SLyjVJqm91ijwJblFLXiMgZ5vJDRSQauBuIV0qViMhnwCjgI7Pc60qpV6ptL95cpgcQBfwkIl2VUlUfxeUApZRLNTOcqoZs+iuzlnHvL/eyInUFozvdy9wfYnATCwtvP4ez29t92y3MgI9HQmkuXPm6kRCCoxssLkcE+nhyRltPzmhbc/IoLLNUNlsdSxxGDeTv1DxyisqrLO/t4UZ0SPUmK+N9bKgv4QHeVR6IU521sJDUu++maPUaIu6ZTtjkyVX+/qw2xbd/pvHmcuORnL1jgnn+ml6c1yW8Wfydas2LIzWH/sAepdQ+ABFZBIwA7JNDPPACgFJqh4jEiUgbu234ikgF4Aek1bK9EcAipVQZsF9E9pgxrHFwnwDw8fEhOzubsDDXflxjbZRSZGdn4+NT/zc6lVhKmP7zdNamr+UfUXcz54e2tGvlxdzb+tMuzO/YgqX58Mm1UHgYxn4LMTU2UbqcAG8PurUNpFvbmu/mLiqzmEmj2K75yvi8NO0Q2dWSh5f70ZrH8bWOyIpCSu+/m7K9e4n8178IGXlNZTmbTfFj4iFeX7aL3RmFdI8M4oNbExjavXWT/tvUmjdHkkM0kGz3OQU4p9oyfwIjgZUi0h9oD8QopTaKyCtAElACLFVKLbUrN01EbgU2APcrpY6Y21tbbXvHfUUVkUnAJDC6g1YXExNDSkoKmZmZx81ranx8fIiJianXdRZXFHPXz3ex/tB6BgXfyYLlUQzs1Ip3R59NsJ/nsQUtZbB4DBxOhJsXN5nE4Ah/bw+6tgmka5uak0dxuVnzqFbrSDlSwrJth8kqNJJHbMFhnl39PkEVJbw/dAoFOVHE/HercWOgnxcL/0hiW3o+nVsH8M7ovgzv0fakNRBNcwWOJIea/oqrt3O8CLwpIlswritsBizmtYgRQAcgF/hcRMYopT4B3gWeNdf1LPAqMN7B7aGUmg3MBqO3UvX5np6edOjQwYHda3mKKoqYunwqmzM2c4b7JH5YG8sNCTE8d3W1Hkk2G3w1Gfb/BlfPMga7a0H8vDzo0iaQLidIHiXlVpJ/X03FjKexenmw6o7ncQ+IpPBICT9tzyCrsAyAuDA/3rixD//oHYW7TgpaE+FIckgBYu0+x1CtaUgplQ+MAxCjnrzffF0K7FdKZZrzvgQGAp8opQ4fLS8i7wPfObo97dQVlhdyx093sDXrL9qUjmfdvg48NLwbd5zfqWoTh1Lw4wxI/AqGPQN9bnJe0C6q4pfl2B58EJ/oaGLff58zY6pWcEsrrBzOLyU6xBcPd9cZL0vTHOHIX+x6oIuIdBARL4yLxd/YLyAiIeY8MHom/W4mjCRggIj4mUljKLDdLBNpt4prgL/N998Ao0TEW0Q6AF2Adae2e5q9/PJ8Ji+bzF9Zf+F7ZCwHk7oy8+a+3HlB5+Pbvle+BuveM0ZCHXi3cwJ2YTnz55N6zz349OhB+wWf4hVz/MV5H0932of568SgNUm11hyUUhYRmQYswejK+qFSKlFEppjzZwHdgfkiYsW4UD3BnPeHiHwBbAIsGM1Ns81V/1tE+mA0GR0AJptlEs1eTdvMMlNPpaeSVlVeWR6Tl01mR85OrIduwVLek4WTEujbroY+9Zvmw/JnoNcNMOzZFjPekSOUzUbGK6+S8+GHBA67mKiXX8atAToLaJqzNds7pLVjjpQeYdKySew+sofi5DG0801g7m39iG3ld/zCO76HxaOh44Vw0yLwcK2HDjmTrbyc9BmPkP/994SOHk2bRx9B3Ov33g5Na0z6DukWLLskm9uX3s6+3AMUJN3KuZEDmTm6L8G+nscvnLQWvhgHkX3ghvk6Mdix5ueTMu0uiteto/UD99NqwgTdDVVr1nRyaMaySrIYv2QCB/NSKDw4lht6XsgzI3riWVMbeMZ2WHAjBEXD6M/BW48AelRFejrJkyZTduAAUS//m+B//MPZIWlag9PJoZnKKM7gth/Gk1JwiKKksTx0/hVMOq9jzd9281KMm9w8fOCWr8DfeaPCuprSnbtInjQJW1ER7Wa/h/+55zo7JE1rFDo5NEOHig5x6/fjSC/MpCJ1PG9fcw2X9YqseeHiHGNYjLICGPdDi3sc58kUrf2DlLvuws3Xl/affoJPt27ODknTGo1ODs1MWmEao7+7jaySI3hmTuHjsdfRJzak5oXLi2HBDXDkANzyJbTt2ZihurS8//2P9BmP4Nm+He1mz8az2hP8NK2508mhGUkuSOamb28jt7SA8MK7+OT264gJraFHEoC1Aj6/DVI2GBef4wY3aqyuSilFztyPyPj3v/FLSCBm5tu4B+vnKmgtj04OzcSB3IPc+O1YispL6MYDzJ10DUE+NfRIAuPu52+nw+4lcMVrEH9V4wbropTVyuGXXuLI/I8JvGw4US++iJu3t7PD0jSn0MmhGdiRvZfR391GqaWCC4Kf4M1rLj/5XbnLn4Ytn8IFj0C/CY0XqAuzlZWR9uBDFCxdSquxY2n98EOIm76zWWu5dHJo4tanbuf2pROpsNm4pf0LPDz0gpP3v1/zDqx8HRLGw/kPN16gLsyam0vy1GmUbNxI6xkPE3bbbc4OSdOcTieHJmzZni3c//sd2JTwcO83uDWh+kjq1fz1BSx5BLr/Ay5/RQ+LAVSkppI0aTIVSUlEv/4aQZdd5uyQNM0l6OTQRC3YsoYXNt0DypPXhrzLJV17nbzA3p/hqynQfhCM/KDeH+nZFJVu307ypMnYysqInfMB/v37OzskTXMZOjk0Qa/+tpy5ex/BQ3x4/9I59IvucvICqZtg8S0Q0Q1GLQBPPVBc4apVpN49HbfAQOI+/QTvLrUcQ01rYXRyaEJsNsUD337L0uxn8XYLYMGVc+kWHnfyQtl74dPrwa8VjP4CfEMaI1SXlvf116Q99jjeHTsS+/5sPNu0qb2QprUwOjk0ESXlVsYv+oy/LK/i7xnMFyPmExt0/DMEqig4DB9fAygY8xUEneAu6RZCKUX2+x+Q+dpr+A0YQMx/3sI9sOanvGlaS6eTQxOQUVDKmE8Wkub9NqE+4Xw+Yh5t/duevFBpnjFeUlEWjP0Wwjs3TrAuSlmtHHruOXIXLiLoyiuJ+tfziJcedVbTTkQnBxe341A+YxcspCj0Pdr4RbLoHx8R4Rdx8kIVpbBoNGRuh5sXQ8zZjROsi7KVlJD6wIMULl9O2MQJRNx3n76HQdNqoZODC/t1ZwbTvlqMtJlLbGAMH1/xIeG+tYyYarPCV5PgwAoY+T50vrhxgnVRliNHSJlyByVbt9Lm8cdpNWa0s0PStCZBJwcX9fHagzzz05f4xMynY3Accy+bQyufVicvpBT88BBs+xoueR7OvKFxgnVR5cnJJN8+iYr0dKLffIOgSy5xdkia1mTo5OBirDbFC99vZ+6W/+Efs4AurToz55L3CfEJqb3w7y/D+g9g4N0wcFqDx+rKSv5OJHnyZJTFQru5H+LXt6+zQ9K0JkUnBxdSXG5h+qIt/JK8HP/YhcSHncF7w94j2NuBUUE3fgS/PA+9b4KLn27wWF1Z4YoVpEy/B4+QEGI/no93x47ODknTmhx9Vc5FHM4v5Yb31vBryk/4xyykV0QP3r/kfccSw/bv4Lt7ofMwuOo/0IIvtub+90uSp9yBV1x72i9aqBODpp0ih84iIjJcRHaKyB4RmVHD/FAR+UpEtorIOhHpaTfvXhFJFJG/RWShiPiY018WkR1mma9EJMScHiciJSKyxXzNqqd9dVnb0vK5euYq9peswi96Ib1bn8l7F79HoJcDffAPrIIvxkNUX7hhHrifYJjuZk4pRebMmaQ/9hj+AwbQfv7HeLZu7eywNK3JqjU5iIg7MBO4DIgHbhKR+GqLPQpsUUqdCdwKvGmWjQbuBhKUUj0Bd2CUWWYZ0NMsswt4xG59e5VSfczXlFPeuybglx0ZXD9rNeU+63Fvs5C+bfoy6+JZBHgF1F74cCIsvAlC2sHNn4GXf8MH7IKUxcKhJ58i6z9vEzxiBLGz3sU9oGUeC02rL47UHPoDe5RS+5RS5cAiYES1ZeKB5QBKqR1AnIgcHZPAA/AVEQ/AD0gzl1uqlLKYy6wFYk5rT5qg+WsOMGHeesIj/6Si1QL6R/bjnYvfwc/zBE9vs5ebZNzk5uVnPOLTP6zhA3ZBtuJiUqZOI/fzzwmbMpnIF19APFtm7UnT6pMjySEaSLb7nGJOs/cnMBJARPoD7YEYpVQq8AqQBKQDeUqppTVsYzzwg93nDiKyWUR+E5EhNQUlIpNEZIOIbMjMzHRgN1yH1aZ4+ttEnvw6kZ5nbCPH7xMGRg3k7YvextfDt/YVFGXDxyOhohjGfGnUHFogS3Y2B8feRuGKFbT95z9pfc89J3+WhaZpDnMkOdT036aqfX4RCBWRLcBdwGbAIiKhGLWMDkAU4C8iY6qsXOQxwAJ8ak5KB9oppc4C7gMWiEjQcQEoNVsplaCUSoiIqOWOYRdSVGZh0vwNzF11gPPO3sV+5nNezHm8edGb+Hg4MFpqeREsuB7ykuGmRdCmegtfy1B+8CAHbrqZst27iXn7P4SOutHZIWlas+JIV9YUINbucwxm09BRSql8YByAGF/d9puvS4H9SqlMc96XwEDgE/PzWOBKYKhSSpnrKgPKzPcbRWQv0BXYcGq76DoO5ZUyYd56tqfnM+K8vfyc+SEXxl7IK+e/gpe7A+P8WCvgs1shbTPc+Am0H9jwQbugkq1bSZ5sXIpq/9FcfPv0cW5AmtYMOVJzWA90EZEOIuKFcUH5G/sFRCTEnAcwEfjdTBhJwAAR8TOTxlBgu1lmOPAwcJVSqthuXRHmRXBEpCPQBdh3OjvpChLT8rh65ioOZBVx8yX7+TnzfYa1H8arF7zqWGKw2eDrabDnJ7jydTjjioYP2gUV/PILB28di1tAAHELF+jEoGkNpNbkYF40ngYswTixf6aUShSRKSJytCdRdyBRRHZg9Gqabpb9A/gC2AT8ZW5vtlnmbSAQWFaty+p5wFYR+dMsO0UplXP6u+o8y7cf5vpZa3ATuPnS3Xyd9B7D44bz0nkv4enm4MXTn56CrYvgwsfh7NsaNF5XdWTxZ6RMnYZ3587ELVyAV1ycs0PStGZLzNacJi0hIUFt2OCarU5zV+3n2e+20SMqmPPP2cy87bO5ouMVPDfoOTzcHLxBffV/YOnj0O92uPzlFvfsZ6UUmW+9Rfa7s/A//zxiXnsNN3/dVVXTTpeIbFRKJdQ0Tw+f0UAsVhvPfreNeWsOMqx7a7r3WM3cxA8Y0WkETw98GndHn+H852IjMcRfDZe91PISQ0UF6U8+Rd5XXxF83bVE/vOfiIf+s9W0hqb/yxpAYZmFuxZs4pedmUwcHIdf26XMTfyQa7tcy5PnPombODi8xe6f4Os7IW4IjJwNjiaUZsJaWETqPfdQtHIl4dOmET71Tt1VVdMaiU4O9Sw9r4TxH21g1+ECnh3Rg0MenzM3cT43druRR8951PHEkLLR6JnUujuM+hQ8vBs2cBdjycwkefIUSnfuJPK5Zwm57jpnh6RpLYpODvXor5Q8JsxbT3G5lTljE1iTO4cF2xYwuvtoHu73sOPferN2G/cy+IfD6P+CjwOD7zUjZfv2k3z77Vhycoh9ZyYB55/v7JA0rcVpucN31rOliYe44b01eLq78fmUAazIeY8FOxYwNn5s3RJDfrpx9zMCt3wFgW1qLdKcFG/azMGbbsJWWkr7+fN1YtA0J9HJ4TQppfhgxT4mf7KRrm0C+O+dA1i8/3U+2/UZE3pO4P6E+x1PDCW5xnhJJTkw5gsI69Sgsbua/GXLSBo3DveQEOIWLcS3V8/aC2ma1iB0s9JpsFht/PPbRD5Zm8TwHm155fpevLDhab7Z+w2Tz5zM1D5THU8MFaWw6GbI2gWjP4Oosxo2eBeT8+mnHH7ueXzO7EXsu+/i0aqWR6JqmtagdHI4RQWlFUxbsJnfdmUy+byO3H9JZ55Y/Tjf7/+eqX2mMqV3HUYat1nhvxPg4Cq4dg50uqjhAncxymYj8/XXyX7/AwIuuojoV1/BzdeBwQc1TWtQOjmcgtTcEiZ8tJ7dGYW8MLIX1yVE8siKR1hyYAnT+05nYq+Jjq9MKfjffbDjOxj+EvRqOb1yVHk5aY89Tv633xIy6kbaPv64vodB01yE/k+so60puUyYt4HScivzxvXnnI7BPPTbQ/yU9BMPJDzA2B5j67bCX180nv88+F4Y0Kyfa1SFtaCAlLvvpnjNWiLuuYewyZP0PQya5kJ0cqiDH/8+xD2LNxPm782nd55DXLg39/12H78m/8qM/jMY3X103Va4fg789iL0GQNDn2qQmF1RxeHDJE+aTNnevUS++AIhV1/t7JA0TatGJwcHKKV4f8U+XvhhB71jQnj/1gSC/OCeX+5hReoKHj/ncW48o47PE9j2Nfzvfug6HP7xZosZFqNszx6Sbp+ELS+P2FmzCBg8yNkhaZpWA50calFhtfHk14ksXJfEFb0iefWG3iAV3P3zdNakreGpc5/iuq51vE6wfwX8dyLE9IPr5oJ7y/g1FK9fT/LUaYi3F+0/+Rif+Jb5oCJNawpaxlnpFOWXVjD1002s2J3FnRd04oFLulFqLeHu5Xez7tA6nhn0DFd3vrpuKz30l9FlNbQD3LzYeAZ0C5D/44+kPfgQnrGxxM6ejVdM9SfNaprmSnRyOIHknGImzFvPvswi/n3tmdzQL5biimLuXH4nmzM28/zg5/lHp3/UbaVHDhg3uXkHwi1fgl/L6MufM28eh198Cd+zziL2nZm4h4Q4OyRN02qhk0MNtiTnMnHeesosNuaN78+gzuEUlhdy5/I72Zq5lReHvMhlHS6r20qLsoxhMSxlMP5rCI5pmOBdiLLZyPj3y+R89BGBw4YR9fK/cfNx4DnZmqY5nU4O1Xz/Vzr3Lt5C6yBvFk0aQOfWgeSX53PHT3ewLWsbL5//MsPaD6vbSssK4dPrID8Nbv3aGGm1mbOVl5M+Ywb53/9A6JgxtHlkBuLesoYc17SmTCcHk1KKWb/t46Ufd9C3XQizb00gPMCbvLI8Ji+bzM4jO3n1gle5qF0d7162lMNnt0D6VmPo7XbnNMwOuBBrXh4p0+6ieP16Wj/4AK3Gj9f3MGhaE6OTA0aPpMe/+pvFG5K58sxIXrm+Nz6e7uSW5nL7stvZm7uXNy54g/Nj6zhCqM1mPKxn788wYiZ0q2NTVBNUkZ5O8qRJlB04SNTLLxP8jyudHZKmaaegxSeHvJIK7vx0I6v2ZDPtws7cN6wrbm5CTmkOE5dOJCk/if9c9B8GRdexP75SxuM9//ochj4JZ41pmB1wIaU7d5E8aRK2oiLavT8b/wEDnB2SpmmnyKEhu0VkuIjsFJE9IjKjhvmhIvKViGwVkXUi0tNu3r0ikigif4vIQhHxMae3EpFlIrLb/BlqV+YRc1s7ReTS+tjRmiTnFHPtu6tZtz+Hl687kwcu7Yabm5BVksWEJRNIzk/m7aFv1z0xAKx+C9bOhHOmwOD76j94F1O0di0HRxt3iLf/9BOdGDStias1OYiIOzATuAyIB24Skep3Lz0KbFFKnQncCrxplo0G7gYSlFI9AXdglFlmBrBcKdUFWG5+xlz3KKAHMBx4x4yh3llsCovVxvzx53B9QiwAGcUZjF8yntTCVN65+B0GRJ7CSW7LAlj2JPQYCZe+0Ozvfs777n8k3T4Jz8i2xC1aiE+3bs4OSdO00+RIzaE/sEcptU8pVQ4sAkZUWyYe4wSPUmoHECciRx9h5gH4iogH4AekmdNHAPPM9/OAq+2mL1JKlSml9gN7zBjqXYdwf36673zO7RQGwKGiQ4xfMp7DRYeZdfEs+rXtV/eV7loCX0+DDufDNbPArfk+T0kpRfacOaQ98AB+ffrQ/tNP8YyMdHZYmqbVA0fOXNFAst3nFHOavT+BkQAi0h9oD8QopVKBV4AkIB3IU0otNcu0UUqlA5g/W9dhe4jIJBHZICIbMjMzHdiNmnm4G4cgrTCNcT+OI7skm/eGvUffNn3rvrLk9fDZWGjbE278BDy8TzkuV6esVg4//y8yXn6FwMuGEzvnA9yDgpwdlqZp9cSR5FBTm4iq9vlFIFREtgB3AZsBi3kdYQTQAYgC/EWktiuzjmwPpdRspVSCUiohIiKillWeXEpBCuN+HEdeeR6zh82mT+s+dV9J5k5YcD0EtoXRX4BP8z1R2kpLSb3nXo588gmtbruN6Fdfxc3Ly9lhaZpWjxzprZQCxNp9juFY0xAASql8YByAGB3a95uvS4H9SqlMc96XwEDgE+CwiEQqpdJFJBLIcHR79SkpP4kJSydQYinhg0s+ID7sFAaDy0s17n528zSGxQhoXXuZJsqam0vynVMp2byZNo/MoNXYOj6/QtO0JsGRmsN6oIuIdBARL4yLxd/YLyAiIeY8gInA72bCSAIGiIifmTSGAtvN5b4Bjp5ZxgJf200fJSLeItIB6AKsO7XdO7kDeQcY9+M4yixlzLlkzqklhpIjxnhJpXkw5gto1bH+A3URFYczOHDzaEr/+ovo11/TiUHTmrFaaw5KKYuITAOWYPQ2+lAplSgiU8z5s4DuwHwRsQLbgAnmvD9E5AtgE2DBaG6aba76ReAzEZmAkUSuN8skishn5noswFSllLW+dtiep7snrf1a88ygZ+gS2qXuK6gogQWjIGev0ZQU2bv+g3QRFRkZJI0diyUjg9g5H+Dfv0H6CGia5iJEqeOa85uchIQEtWHDhlMqq5Q6taEdrBZjWIydP8B1H0LPkae0/abAkpnJwbG3UXHoEO3en43f2Wc7OyRN0+qBiGxUSiXUNK/F3yF9SolBKfjfvbDze7js5eadGLKzOThuHBXp6bSb/Z5ODJrWQjTfTvgN6ZfnYdN8OO9BOGeSs6NpMJacHJJuu42K1DRi35uFX79TuO9D07QmqcXXHOrsj9nw+8vQ91a48DFnR9NgLEeOkHTbOMqTU4idNUtfY9C0FkYnh7r4+0v44SHodjlc8XqzHRajMjEcPEjsrHfxH9D8hxnXNK0qnRwcte83+GoytBtgXIB2b56HzpqbS9L4CZTv30/MO+/gf+65zg5J0zQnaJ5nuPqW/icsGg2tOsFNC8HT19kRNQhrXp6RGPbsIeadmQQMPoXRaDVNaxb0Bena5OyDT64Dn2AY81/wDa29TBNkzc8nacJEynbvJubt/xAwZIizQ9I0zYl0zeFkCjOMYTFsFXDbdxB83Ph/zYK1oICkibdTunMnMW+9ScD5dXzinaZpzY5ODidSmm8Mi1FwCMZ+CxHN8xkF1sJCkifeTun27cS8+QaBF17o7JA0TXMBOjnUxFIGi8fA4US4aRHENs/+/dbCIpJvn0RJYiIxb7xO4EUXOTskTdNchE4O1dls8NUU2P8bXD0Lul7i7IgahK2oiOTJkynZupXo114j8OKLnR2SpmkuRF+QtqcU/DgDEr+Ei5+GPjc5O6IGYSsuJnnyFEq2bCH61VcIurR5JkBN006drjnYW/karHsPBkyFQdOdHU2DsBUXkzzlDoo3bSL6lZcJGj7c2SFpmuaCdM3hqE0fw/JnoNf1cMlzzfLuZ1tJCcl33Enxhg1E/fvfBF1+ubND0jTNRemaAxjDbn87HTpdBCPeAbfmlzNtpaWkTJ1K8bp1RL30IsFXXuHskDRNc2HN7yxYV0l/wOe3QeSZcMPH4NH8noVsKysjZeo0itasJfKFfxF81VXODknTNBfXspNDxg5YcAMERcPNn4N3gLMjqne2sjJSpt1F0erVRD73HCFXX+3skDRNawJadnLwCYKos+CWLyEgwtnR1DtbeTkpd99N0YoVRD77DCHXNt+HEmmaVr9a9jWHoCi49f+cHUWDsJWXk3r3dIp++522Tz9NyHXXOTskTdOakJZdc2imVHk5qffcS+Gvv9L2n08ReuMNzg5J07QmRieHZkZVVJB6//0U/vwzbZ54nNBRo5wdkqZpTZBDyUFEhovIThHZIyIzapgfKiJfichWEVknIj3N6d1EZIvdK19E7jHnLbabfkBEtpjT40SkxG7erPrb3ebNSAwPULDsJ9o89hitRo92dkiapjVRtV5zEBF3YCYwDEgB1ovIN0qpbXaLPQpsUUpdIyJnmMsPVUrtBPrYrScV+ApAKXWj3TZeBfLs1rdXKdXnNParxVEWC6kPPkTB0qW0eWQGrW4Z4+yQNE1rwhypOfQH9iil9imlyoFFwIhqy8QDywGUUjuAOBFpU22ZoRgn/YP2E0VEgBuAhacQv4aRGNIeepiCH3+k9UMP0WrsWGeHpGlaE+dIcogGku0+p5jT7P0JjAQQkf5AeyCm2jKjqDkBDAEOK6V2203rICKbReQ3EanxkWQiMklENojIhszMTAd2o3lSVitpMx4h//vvaf3gA4SNH+fskDRNawYcSQ41DTKkqn1+EQg1rxvcBWwGLJUrEPECrgI+r2FdN1E1aaQD7ZRSZwH3AQtEJOi4AJSarZRKUEolREQ0v3sUHKGsVtIeeYT8774j4r77CJswwdkhaZrWTDhyn0MKEGv3OQZIs19AKZUPjIPKZqL95uuoy4BNSqnD9uVExAOjxnG23brKgDLz/UYR2Qt0BTY4tkstg7JaSX/scfK/+ZaIe6YTPul2Z4ekaVoz4kjNYT3QRUQ6mDWAUcA39guISIg5D2Ai8LuZMI6qXjs46mJgh1IqxW5dEebFa0SkI9AF2OfoDrUEymYj/Yknyfu//yP8rmmET5ni7JA0TWtmaq05KKUsIjINWAK4Ax8qpRJFZIo5fxbQHZgvIlZgG1DZviEifhg9nSbXsPqarkOcBzwjIhbACkxRSuXUec+aKWWzceipp8j78kvC77yTiKlTnR2SpmnNkChV/fJB05OQkKA2bGj+rU7KZuPQ08+Qu3gxYVMmEzF9OtIMnzuhaVrjEJGNSqmEmubpO6SbCKUUh5591kgMkybpxKBpWoPSyaEJUEpx+NnnyF24iLCJE4i49x6dGDRNa1A6Obg4pRSH//UCRxYsoNW4cUTcf79ODJqmNTidHFyYUoqMF1/iyMcf02rsrbR+6EGdGDRNaxQ6ObgopRQZL79Czrx5hN5yC61nzNCJQdO0RqOTgwtSSpH56qvkfPghoTffTJtHH9GJQdO0RqWTg4tRSpH5+htkfzCHkJtG0eaJx3Vi0DSt0enk4EKUUmS+9RbZs2cTcsMNtH3iCZ0YNE1zCp0cXEjW2zPJfncWIddfR9t/PoW46V+PpmnOoc8+LiLznXfImjmT4JEjafv00zoxaJrmVPoM5AKyZr1H1lv/IXjECCKffUYnBk3TnE6fhZwsa/b7ZL7xBkFX/YPIfz2PuLs7OyRN0zSdHJwpe84cMl97jaArryTqhRd0YtA0zWXo5OAk2XM/IuPlVwi6/HKiXtSJQdM016KTgxPkzJtHxksvETh8OFH/fgnxcOSBfJqmaY1HJ4dGlvPxJxx+4UUCL7mE6Jf/rRODpmkuSSeHRpSzYAGHn3+egIuHEv3qK4inp7ND0jRNq5FODo3kyKJFHH7mWQIuuoiY117TiUHTNJemk0MjOLL4Mw7982kCLriAmDdeR7y8nB2SpmnaSenk0MByv/iCQ089hf/55xH91ps6MWia1iQ4lBxEZLiI7BSRPSIyo4b5oSLylYhsFZF1ItLTnN5NRLbYvfJF5B5z3j9FJNVu3uV263vE3NZOEbm0nva10eV++RXpTzyJ/5AhxLz1Fm46MWia1kTU2lVGRNyBmcAwIAVYLyLfKKW22S32KLBFKXWNiJxhLj9UKbUT6GO3nlTgK7tyryulXqm2vXhgFNADiAJ+EpGuSinrKe6jU+T+3/+R/thj+J97LjFv/wc3b29nh6RpmuYwR2oO/YE9Sql9SqlyYBEwotoy8cByAKXUDiBORNpUW2YosFcpdbCW7Y0AFimlypRS+4E9ZgxNRt6335L+yKP4DTiHmHdm6sSgaVqT40hyiAaS7T6nmNPs/QmMBBCR/kB7IKbaMqOAhdWmTTOboj4UkdA6bM9l5X33P9IenoFf//7EvvMObj4+zg5J0zStzhxJDjU9bUZV+/wiECoiW4C7gM2ApXIFIl7AVcDndmXeBTphNDulA6/WYXuIyCQR2SAiGzIzMx3YjYaX//33pD30EH4JCcS++w5uvr7ODknTNO2UOHJ7bgoQa/c5BkizX0AplQ+MAxDj0WX7zddRlwGblFKH7cpUvheR94HvHN2eWX42MBsgISHhuOTR2PJ//JHUBx/Ct+9ZxM56Fzc/P2eHpGmadsocqTmsB7qISAezBjAK+MZ+AREJMecBTAR+NxPGUTdRrUlJRCLtPl4D/G2+/wYYJSLeItIB6AKsc3SHnCF/6VJS738A3969affeezoxaJrW5NVac1BKWURkGrAEcAc+VEolisgUc/4soDswX0SswDZgwtHyIuKH0dNpcrVV/1tE+mA0GR04Ot9c92fmeizAVFfuqVTw00+k3nc/vr16ETt7Nm7+/s4OSdM07bSJUk5vkTltCQkJasOGDY2+3YKffyFl+nR84+OJnfMB7gEBjR6DpmnaqRKRjUqphJrm6TukT1HBL0Zi8OnendgP3teJQdO0ZkUnh1NQ+NtvpN49HZ9u3Wj3wfu4BwY6OyRN07R6pZNDHRWuWEnKXXfj3aUL7eZ8gHtQkLND0jRNq3c6OdRB4apVpEydilenTrT7cA7uwcHODknTNK1B6OTgoKI1a0i5cypeHToYiSEkxNkhaZqmNRidHBxQtPYPku+4E6/27Wn30Vw8QkNrL6RpmtaE6eRQi6J160ieMgWv2BidGDRNazF0cjiJ4g0bSJ48Bc/oaNrNnYtHq1bODknTNK1R6ORwAsWbNpE0aTKekZG0/2guHuHhzg5J0zSt0ejkUIPizZtJnng7nq1bG01JERHODknTNK1R6eRQTcmff5I88XbcI8JpN+8jPFu3dnZImqZpjU4nBzslW7eSNGEi7mFhtJ83D8821R9mp2ma1jLo5GAq+etvIzGEhtJ+3kd4tm3r7JA0TdOcRicHoCQxkaQJE3APCjISQ2Rk7YU0TdOasRafHEq3bydp/ATcAvxpN28enlFRzg5J0zTN6Vp0cijduYukceNx8/Oj/fz5eMVEOzskTdM0l9Cik4N7aAg+8fG0n/cRXjExzg5H0zTNZdT6mNDmzLN1a9p9OMfZYWiaprmcFl1z0DRN02qmk4OmaZp2HJ0cNE3TtOM4lBxEZLiI7BSRPSIyo4b5oSLylYhsFZF1ItLTnN5NRLbYvfJF5B5z3ssissMs85WIhJjT40SkxK7MrPrbXU3TNM0RtSYHEXEHZgKXAfHATSISX22xR4EtSqkzgVuBNwGUUjuVUn2UUn2As4Fi4CuzzDKgp1lmF/CI3fr2Hi2nlJpyynunaZqmnRJHag79gT1KqX1KqXJgETCi2jLxwHIApdQOIE5Eqg9MNBTjpH/QXG6pUspizlsL6L6kmqZpLsKR5BANJNt9TjGn2fsTGAkgIv2B9hx/sh8FLDzBNsYDP9h97iAim0XkNxEZUlMBEZkkIhtEZENmZqYDu6FpmqY5ypHkIDVMU9U+vwiEisgW4C5gM3C0VoCIeAFXAZ8ft3KRx8xlPzUnpQPtlFJnAfcBC0Qk6LgAlJqtlEpQSiVE6OctaJqm1StHboJLAWLtPscAafYLKKXygXEAIiLAfvN11GXAJqXUYftyIjIWuBIYqpRS5rrKgDLz/UYR2Qt0BTacKMCNGzdmichBB/blRMKBrNMo31B0XHWj46obHVfdNMe42p9ohiPJYT3QRUQ6AKkYzUM32y9g9jQqNq9JTAR+NxPGUTdRrUlJRIYDDwPnK6WK7aZHADlKKauIdAS6APtOFqBS6rSqDiKyQSmVcDrraAg6rrrRcdWNjqtuWlpctSYHpZRFRKYBSwB34EOlVKKITDHnzwK6A/NFxApsAybYBe4HDAMmV1v124A3sMyobLDW7Jl0HvCMiFgAKzBFKZVzerupaZqm1YVDYysppb4Hvq82bZbd+zUY3/BrKlsMhNUwvfMJlv8v8F9H4tI0TdMahr5D2jDb2QGcgI6rbnRcdaPjqpsWFZeY14E1TdM0rZKuOWiapmnH0clB0zRNO06LSQ4ODB4oIvKWOX+riPR1kbguEJE8u4EIn2ykuD4UkQwR+fsE8511vGqLq9GPl4jEisgvIrJdRBJFZHoNyzjreDkSmzOOmY85SOefZlxP17BMox8zB+Ny1v+kuxgjR3xXw7z6P1ZKqWb/wuiCuxfoCHhhDPcRX22ZyzGG8BBgAPCHi8R1AfCdE47ZeUBf4O8TzG/04+VgXI1+vIBIoK/5PhBjIEmn/33VITZnHDMBAsz3nsAfwABnHzMH43LW/+R9wIKatt0Qx6ql1BwcGTxwBDBfGdYCISIS6QJxOYVS6nfgZPeXOON4ORJXo1NKpSulNpnvC4DtHD/+mLOOlyOxNTrzOBSaHz3NV/XeMY1+zByMq9GJSAxwBfDBCRap92PVUpKDI4MHOrKMM+ICONes5v4gIj0aOCZHOeN4Ocppx0tE4oCzML5x2nP68TpJbOCEY2Y2k2wBMoBlSimXOGYOxAWNf7zeAB4CbCeYX+/HqqUkB0cGD3RkmfrmyDY3Ae2VUr2B/wD/18AxOcoZx8sRTjteIhKAcQPnParq8DHg5ONVS2xOOWZKKasynvUSA/QX8yFhdpxyzByIq1GPl4hcCWQopTaebLEapp3WsWopyaHWwQMdXKbR41JK5R+t5irjTnVPEQlv4Lgc4YzjVStnHS8R8cQ4+X6qlPqyhkWcdrxqi83Zf2NKqVzgV2B4tVlO/Rs7UVxOOF6DgKtE5ABG0/NFIvJJtWXq/Vi1lORQOXigGMOHjwK+qbbMN8Ct5lX/AUCeUird2XGJSFsRY/ApMZ6V4QZkN3BcjnDG8aqVM46Xub05wHal1GsnWMwpx8uR2Jx0zCLk2KOBfYGLgR3VFmv0Y+ZIXI19vJRSjyilYpRScRjniJ+VUmOqLVbvx8qhsZWaOuXY4IHfY1zx34PxONNxLhLXdcAdYgxEWAKMUmb3hIYkIgsxemWEi0gK8BTGxTmnHS8H43LG8RoE3AL8ZbZVg/Ho3HZ2cTnleDkYmzOOWSQwT4zHELsBnymlvnP2/6SDcTnlf7K6hj5WevgMTdM07TgtpVlJ0zRNqwOdHDRN07Tj6OSgaZqmHUcnB03TNO04OjlomqZpx9HJQdM0TTuOTg6apmnacf4fg/rFEqgwEDoAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for d,h in res.items():\n",
    "    plt.plot(h.history['val_acc'],label=str(d))\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this graph, you would probably be able to see the following:\n",
    "* Dropout values in the 0.2-0.5 range, you will see the fastest training the best overall results\n",
    "* Without dropout ($d=0$), you are likely to see less stable and slower training process\n",
    "* High dropout (0.8) makes things worse"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "86193a1ab0ba47eac1c69c1756090baa3b420b3eea7d4aafab8b85f8b312f0c5"
  },
  "kernelspec": {
   "display_name": "Python 3.9.5 64-bit ('base': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}