GP Regression using GPyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "artificial-shore",
   "metadata": {},
   "source": [
    "# Gaussian Process (GP) Regression using GPyTorch\n",
    "\n",
    "Author: [Richard Cornelius Suwandi](https://richardcsuwandi.github.io)\n",
    "\n",
    "In this tutorial, I'm going to demonstrate how to perform Gaussian Process (GP) regression using GPyTorch. \n",
    "[GPytorch](https://github.com/cornellius-gp/gpytorch) is a Gaussian process library implemented using PyTorch that is designed for creating scalable and flexible, and modular GP models. You can learn more about GPyTorch on their [official website](https://gpytorch.ai)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "behavioral-dietary",
   "metadata": {},
   "source": [
    "## Setup\n",
    "Let's first import the necessary libraries for this tutorial. If you haven't installed `gpytorch`, you can install it either using pip or conda:\n",
    "\n",
    "```bash\n",
    "pip install gpytorch\n",
    "conda install gpytorch -c gpytorch\n",
    "```\n",
    "You can also check the requirements and installation instructions on their website [here](https://gpytorch.ai/#install)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "standing-problem",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the necessary libraries\n",
    "import math\n",
    "from matplotlib import pyplot as plt\n",
    "import torch\n",
    "import gpytorch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spare-tunnel",
   "metadata": {},
   "source": [
    "## Generating the data\n",
    "Next, we need to generate a training data for our model. We will be modeling the following function:\n",
    "\n",
    "\\begin{align}\n",
    "    y = \\sin{(2 \\pi x)} + \\epsilon, \\enspace \\epsilon \\sim \\mathcal{N}(0, 0.04)\n",
    "\\end{align}\n",
    "where $\\epsilon$ is the Gaussian noise. We will evaluate the above function on 15 equally-spaced points from $[0,1]$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "friendly-factor",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1a5432e0bb0>]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD5CAYAAAAqaDI/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYPElEQVR4nO3dcXCc9X3n8fen5oy0k24xsYp0YGO3cdvQpDXtHkmmM1USDHEyNzZc09TImZg7UmdD6c00Ry50uJl0pHTGuXAluTmjoBI3JNMLTjiX6CbJ+AiBy/gO5bxuKGAyFEcKwe5KqJgwc2Mv1Ph7f+xjbpEla8Xz7K5Xz+c1s6Pn+T2/fZ7vg4w+ep7fs/opIjAzs/z6uU4XYGZmneUgMDPLOQeBmVnOOQjMzHLOQWBmlnMOAjOznLsgi51I2gP8S+D5iHjbPNsFfAH4AHACuDEi/jbZtgP4D0nXz0TEvYsdb/Xq1bFu3bosSjczy41Dhw79Y0T0zW3PJAiALwP/BfjKAtvfD2xIXu8ARoF3SLoY+DRQAgI4JGk8Il4818HWrVtHpVLJqHQzs3yQ9Ox87ZncGoqI7wPHz9FlK/CVqJsALpI0ALwPeDAijic//B8ENmdRk5mZNaddYwSXAs81rB9N2hZqNzOzNumawWJJOyVVJFVmZ2c7XY6Z2bLRriA4BqxpWL8saVuo/SwRMRYRpYgo9fWdNdZhZmZvULuCYBz4iOreCbwUEVVgP3CtpFWSVgHXJm1mZtYmmQSBpK8BjwK/KumopJsklSWVky7fBiaBI8BfAjcDRMRxYAQ4mLyGkzZrkWq1yuDgINPT050uxczOE5k8PhoRNyyyPYA/WmDbHmBPFnXY4kZGRjhw4ADDw8PcddddnS7HzM4D6sb5CEqlUvhzBEvT29tLrVY7q72np4eTJ092oCIzazdJhyKiNLe9a54asnQmJycZGhqiUCgAUCgU2L59O1NTUx2uzMw6zUGQEwMDAxSLRWq1Gj09PdRqNYrFIv39/Z0uzcw6zEGQIzMzM5TLZSYmJiiXyx4wNjPAYwRmZrnhMQJrKT+Wata9HASWicbHUs2su/jWkKXix1LNuodvDVlL+LFUs+7nILBU/FiqWfdzEFhqfizVrLt5jOA8Vq1W2bZtG3v37vVv2GaWmscIupCfxDGzdvAVwXnIT+KYWSv4iqCL+EkcM2snB8F5yE/imFk7OQjOU34Sx8zaJZMxAkmbgS8AK4B7ImLXnO13Au9JVgvAL0bERcm2V4Enkm0/jYgtix1vuY8RmJm1wkJjBKmnqpS0AtgNXAMcBQ5KGo+Ip870iYg/aej/x8CVDbs4GREb09ZhZmZvTBa3hq4CjkTEZES8AtwHbD1H/xuAr2VwXDMzy0AWQXAp8FzD+tGk7SySLgfWA99raO6RVJE0Iem6hQ4iaWfSrzI7O5tB2WZmBu0fLN4G3B8Rrza0XZ7csxoCPi/pl+d7Y0SMRUQpIkp9fX3tqNXMLBeyCIJjwJqG9cuStvlsY85toYg4lnydBB7h9eMHZmbWYlkEwUFgg6T1klZS/2E/PreTpF8DVgGPNrStknRhsrwa+B3gqbnvNTOz1kn91FBEnJJ0C7Cf+uOjeyLisKRhoBIRZ0JhG3BfvP551bcCd0s6TT2UdjU+bWRmZq3nvzVkZpYT/ltDZmY2LweBmVnOOQjMzHLOQWBmlnMOAjOznHMQmJnlnIPAzCznHARmZjnnIDAzyzkHgZlZzjkIzMxyzkFgZpZzDgIzs5xzEJiZ5ZyDwMws5xwEZmY55yAwM8u5TIJA0mZJT0s6Ium2ebbfKGlW0mPJ66MN23ZIeiZ57ciiHjMza17qOYslrQB2A9cAR4GDksbnmXt4b0TcMue9FwOfBkpAAIeS976Yti4zM2tOFlcEVwFHImIyIl4B7gO2Nvne9wEPRsTx5If/g8DmDGoyM7MmZREElwLPNawfTdrm+j1Jj0u6X9KaJb4XSTslVSRVZmdnMyjbzMygfYPF/x1YFxG/Qf23/nuXuoOIGIuIUkSU+vr6Mi/QzCyvsgiCY8CahvXLkrbXRMQLEfFysnoP8NvNvtfyrVqtMjg4yPT0dKdLMVu2sgiCg8AGSeslrQS2AeONHSQNNKxuAX6ULO8HrpW0StIq4NqkzQyAkZERDhw4wPDwcKdLMVu2Uj81FBGnJN1C/Qf4CmBPRByWNAxUImIc+LeStgCngOPAjcl7j0saoR4mAMMRcTxtTdb9ent7qdVqr62Pjo4yOjpKT08PJ0+e7GBlZsuPIqLTNSxZqVSKSqXS6TKsharVKrfeeisPPPAAJ06coFAocP3113PHHXfQ39/f6fLMupKkQxFRmtvuTxbbeWlgYIBisUitVqOnp4darUaxWHQImLWAg8DOWzMzM5TLZSYmJiiXyx4wNmsR3xoyM8sJ3xoyM7N5OQgy4GfdzaybOQgy4GfdzaybeYwghbnPup/hZ93N7HzkMYIWmJycZGhoiEKhAEChUGD79u1MTU11uDIzs+Y5CFLws+5mthw4CFLys+5m1u08RmC5U61W2bZtG3v37vXVm+WKxwjMEn7Ky+z1fEVgueGnvCzvfEVgueenvMzm5yCw3PBTXmbzcxBYrvgpL7OzZTJGIGkz8AXqM5TdExG75mz/BPBR6jOUzQL/JiKeTba9CjyRdP1pRGxZ7HgeIzAzW7qFxghST1UpaQWwG7gGOAoclDQeEU81dPshUIqIE5I+DvxH4A+SbScjYmPaOszM7I3J4tbQVcCRiJiMiFeA+4CtjR0i4uGIOJGsTgCXZXBcMzPLQBZBcCnwXMP60aRtITcB32lY75FUkTQh6boM6jEzsyVIfWtoKSR9GCgBgw3Nl0fEMUm/BHxP0hMR8eN53rsT2Amwdu3attRrZpYHWVwRHAPWNKxflrS9jqRNwO3Aloh4+Ux7RBxLvk4CjwBXzneQiBiLiFJElPr6+jIo28zMIJsgOAhskLRe0kpgGzDe2EHSlcDd1EPg+Yb2VZIuTJZXA78DNA4ym5lZi6W+NRQRpyTdAuyn/vjonog4LGkYqETEOPA54E3ANyTB/39M9K3A3ZJOUw+lXXOeNjIzsxbz3xoyM8sJ/60hMzObl4PAzCznHARmZjnnIDAzyzkHgZlZzjkIzMxyzkFgZpZzDgIzs5xzEJiZ5ZyDwMws5xwEZmY55yAwM8s5B4GZWc45CMzMcs5BYGaWcw4CM7MuUK1WGRwcZHp6OvN9OwjMzLrAyMgIBw4cYHh4OPN9ZxIEkjZLelrSEUm3zbP9Qkl7k+0/kLSuYdufJu1PS3pfFvWYmS0Xvb29SGJ0dJTTp08zOjqKJHp7ezM7RuogkLQC2A28H7gCuEHSFXO63QS8GBFvAe4EPpu89wrqk93/OrAZuCvZn5mZAZOTkwwNDVEoFAAoFAps376dqampzI6RxRXBVcCRiJiMiFeA+4Ctc/psBe5Nlu8HrlZ9FvutwH0R8XJETAFHkv2ZmRkwMDBAsVikVqvR09NDrVajWCzS39+f2TGyCIJLgeca1o8mbfP2iYhTwEvAm5t8LwCSdkqqSKrMzs5mULaZWXeYmZmhXC4zMTFBuVzOfMD4gkz31kIRMQaMAZRKpehwOWZmbbNv377Xlnfv3p35/rO4IjgGrGlYvyxpm7ePpAuAXwBeaPK9ZmbWQlkEwUFgg6T1klZSH/wdn9NnHNiRLH8Q+F5ERNK+LXmqaD2wAfg/GdRkZmZNSn1rKCJOSboF2A+sAPZExGFJw0AlIsaBLwFflXQEOE49LEj6fR14CjgF/FFEvJq2JjMza57qv5h3l1KpFJVKpdNlmJl1FUmHIqI0t92fLDYzyzkHgZlZzjkIzMxyzkFgZpZzDgIzs5xzEJiZ5ZyDwMws5xwEZmY55yAwM8s5B4GZWc45CMzMcs5BYGaWc7kKgmq1yuDgYOaz+5iZdbNcBcHIyAgHDhxgeHi406WYmZ03chEEvb29SGJ0dJTTp08zOjqKJHp7eztdmi0jvuK0bpWLIJicnGRoaIhCoQBAoVBg+/btTE1NdbgyW058xWndKlUQSLpY0oOSnkm+rpqnz0ZJj0o6LOlxSX/QsO3LkqYkPZa8NqapZyEDAwMUi0VqtRo9PT3UajWKxSL9/f2tOJzljK84rdulvSK4DXgoIjYADyXrc50APhIRvw5sBj4v6aKG7Z+MiI3J67GU9SxoZmaGcrnMxMQE5XLZl++WGV9xWrdLO2fxVuDdyfK9wCPApxo7RMTfNyz/g6TngT7gZymPvST79u17bXn37t3tPLQtc77itG6X9orgkoioJsvTwCXn6izpKmAl8OOG5j9PbhndKenClPWYdYSvOK2bLTp5vaTvAvP9anM7cG9EXNTQ98WIOGucINk2QP2KYUdETDS0TVMPhzHgxxEx70ibpJ3AToC1a9f+9rPPPnvOus3M7PUWmrx+0VtDEbHpHDudkTQQEdXkh/rzC/QrAt8Cbj8TAsm+z1xNvCzpr4Bbz1HHGPWwoFQqnTu9zMysaWlvDY0DO5LlHcA353aQtBL4G+ArEXH/nG0DyVcB1wFPpqzHzMyWKG0Q7AKukfQMsClZR1JJ0j1Jnw8BvwvcOM9jon8t6QngCWA18JmU9ZiZ2RItOkZwPiqVSlGpVDpdhplZV1lojCAXnyw2M7OFOQjMzHLOQWBmlnMOAjOznHMQmJnlnIPAzCznHARmZjnnIDAzyzkHgZlZzjkIzMxyzkFgZpZzDgIzs5xzEJiZ5ZyDwMws5xwEZmY55yAwM8s5B4GZWc6lCgJJF0t6UNIzyddVC/R7tWGayvGG9vWSfiDpiKS9yfzGZmZtUa1WGRwcZHp6utOldFTaK4LbgIciYgPwULI+n5MRsTF5bWlo/yxwZ0S8BXgRuCllPWZmTRsZGeHAgQMMDw93upSOSjVnsaSngXdHRFXSAPBIRPzqPP3+b0S8aU6bgFmgPyJOSXoX8GcR8b7Fjus5i80sjd7eXmq12lntPT09nDx5sgMVtUer5iy+JCKqyfI0cMkC/XokVSRNSLouaXsz8LOIOJWsHwUuTVmPmdmiJicnGRoaolAoAFAoFNi+fTtTU1MdrqwzFg0CSd+V9OQ8r62N/aJ+abHQ5cXlSQoNAZ+X9MtLLVTSziRMKrOzs0t9u1nX8n3s7A0MDFAsFqnVavT09FCr1SgWi/T393e6tI5YNAgiYlNEvG2e1zeBmeSWEMnX5xfYx7Hk6yTwCHAl8AJwkaQLkm6XAcfOUcdYRJQiotTX17eEUzTrbr6P3RozMzOUy2UmJiYol8uZBm23hXfaMYLPAS9ExC5JtwEXR8S/n9NnFXAiIl6WtBp4FNgaEU9J+gbw3yLiPklfBB6PiLsWO67HCCwP8nofezm4+eabufvuu/nYxz7GXXct+iOtbVo1RrALuEbSM8CmZB1JJUn3JH3eClQk/R3wMLArIp5Ktn0K+ISkI9THDL6Ush6zZcP3sbtPb28vkhgdHeX06dOMjo4iid7e3k6Xdk6pgiAiXoiIqyNiQ3IL6XjSXomIjybL/zsi3h4Rv5l8/VLD+ycj4qqIeEtE/H5EvJzudMyWD9/H7j7dGt7+ZLHZeayV97Ete90a3hcs3sXMOmXfvn2vLe/evbuDlVizzoT3zp07GRsbo1qtLv6mDks1WNwpHiw2M1u6Vg0Wm5lZl3MQmOVUtz3rbq3jIDDLKX9Qzc7wGIFZzviDavnlMQIzA7r3WXdrHQeBWc5067Pu1joOArMc8gfVrJHHCMzMcsJjBGZmNi8HgZlZzjkIzMxyzkFgZpZzDgIzs5xzEJiZ5VyqIJB0saQHJT2TfF01T5/3SHqs4VWTdF2y7cuSphq2bUxTj5mZLV3aK4LbgIciYgPwULL+OhHxcERsjIiNwHuBE8D/aOjyyTPbI+KxlPWYmdkSpQ2CrcC9yfK9wHWL9P8g8J2IOJHyuGZmlpG0QXBJRJyZh20auGSR/tuAr81p+3NJj0u6U9KFC71R0k5JFUmV2dnZFCWbmVmjRYNA0nclPTnPa2tjv6j/rYoF/16FpAHg7cD+huY/BX4N+BfAxcCnFnp/RIxFRCkiSn19fYuVbWZmTVp08vqI2LTQNkkzkgYiopr8oH/+HLv6EPA3EfFPDfs+czXxsqS/Am5tsm4zM8tI2ltD48COZHkH8M1z9L2BObeFkvBAkqiPLzyZsh4zM1uitEGwC7hG0jPApmQdSSVJ95zpJGkdsAb4n3Pe/9eSngCeAFYDn0lZj5mZLdGit4bOJSJeAK6ep70CfLRh/SfApfP0e2+a45uZWXr+ZLGZWc45CMzMcs5BYGaWcw4CM7OccxCYmeWcg8DMLOccBGZmOecgMDPLOQeBmVnOOQjMzHLOQWBmlnMOAjOznHMQmJnlnIPAzM571WqVwcFBpqenO13KsuQgMLPz3sjICAcOHGB4eLjTpSxLqk813F1KpVJUKpVOl2FmLdbb20utVjurvaenh5MnT3agou4m6VBElOa2p7oikPT7kg5LOi3prJ039Nss6WlJRyTd1tC+XtIPkva9klamqcfMOivrWziTk5MMDQ1RKBQAKBQKbN++nampqUz2b3Vpbw09Cfwr4PsLdZC0AtgNvB+4ArhB0hXJ5s8Cd0bEW4AXgZtS1mNmHZT1LZyBgQGKxSK1Wo2enh5qtRrFYpH+/v5M9m91qYIgIn4UEU8v0u0q4EhETEbEK8B9wNZkwvr3Avcn/e6lPoG9mXWZ3t5eJDE6Osrp06cZHR1FEr29van3PTMzQ7lcZmJignK57AHjFkg1Z3GTLgWea1g/CrwDeDPws4g41dB+1rzGZnb+m5yc5NZbb+WBBx7gxIkTFAoFrr/+eu64447U+963b99ry7t37069PzvbokEg6bvAfNdht0fEN7MvacE6dgI7AdauXduuw5pZE3wLp7stGgQRsSnlMY4BaxrWL0vaXgAuknRBclVwpn2hOsaAMag/NZSyJjPL2JlbODt37mRsbIxqtdrpkqxJ7bg1dBDYIGk99R/024ChiAhJDwMfpD5usANo2xWGmWXLt3C6V9rHR6+XdBR4F/AtSfuT9n8u6dsAyW/7twD7gR8BX4+Iw8kuPgV8QtIR6mMGX0pTj5mZLZ0/UGZmlhMt+UCZmZl1PweBmVnOOQjMzHLOQWBmlnNdOVgsaRZ49g2+fTXwjxmW0w18zvngc17+0p7v5RHRN7exK4MgDUmV+UbNlzOfcz74nJe/Vp2vbw2ZmeWcg8DMLOfyGARjnS6gA3zO+eBzXv5acr65GyMwM7PXy+MVgZmZNVi2QbDQPMkN2y9M5kk+ksybvK4DZWaqiXP+hKSnJD0u6SFJl3eiziwtds4N/X5PUpxrbu1u0Mz5SvpQ8n0+LOm/trvGrDXx73qtpIcl/TD5t/2BTtSZJUl7JD0v6ckFtkvSf07+mzwu6bdSHTAilt0LWAH8GPglYCXwd8AVc/rcDHwxWd4G7O103W045/cAhWT543k456Tfz1OfV3sCKHW67hZ/jzcAPwRWJeu/2Om623DOY8DHk+UrgJ90uu4Mzvt3gd8Cnlxg+weA7wAC3gn8IM3xlusVwbzzJM/ps5X6PMlQnzf56mQe5W616DlHxMMRcSJZnaA+GVA3a+b7DDACfBaotbO4FmjmfP8Q2B0RLwJExPNtrjFrzZxzAMVk+ReAf2hjfS0REd8Hjp+jy1bgK1E3QX2Sr4E3erzlGgTzzZM8dz7k1/pEfc6El6jPidCtmjnnRjdR/42imy16zskl85qI+FY7C2uRZr7HvwL8iqT/JWlC0ua2VdcazZzznwEfTuZG+Tbwx+0praOW+v/7ObVjhjI7z0j6MFACBjtdSytJ+jngL4AbO1xKO11A/fbQu6lf8X1f0tsj4medLKrFbgC+HBH/SdK7gK9KeltEnO50Yd1iuV4RLDRP8rx9JF1A/ZLyhbZU1xrNnDOSNgG3A1si4uU21dYqi53zzwNvAx6R9BPq91LHu3jAuJnv8VFgPCL+KSKmgL+nHgzdqplzvgn4OkBEPAr0UP+bPMtZU/+/N2u5BsFr8yRLWkl9MHh8Tp9x6vMkQ33e5O9FMgrTpRY9Z0lXAndTD4Fuv3cMi5xzRLwUEasjYl1ErKM+LrIlIrp1ertm/l0/QP1qAEmrqd8qmmxjjVlr5px/ClwNIOmt1INgtq1Vtt848JHk6aF3Ai9FRPWN7mxZ3hqKiFOSzsyTvALYExGHJQ0DlYgYpz4/8leT+ZKPU/8H1rWaPOfPAW8CvpGMi/80IrZ0rOiUmjznZaPJ890PXCvpKeBV4JMR0bVXuk2e878D/lLSn1AfOL6xy3+pQ9LXqAf66mTs49PAPwOIiC9SHwv5AHAEOAH861TH6/L/XmZmltJyvTVkZmZNchCYmeWcg8DMLOccBGZmOecgMDPLOQeBmVnOOQjMzHLOQWBmlnP/D0xlqKYLn0h3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The training data is 15 equally-spaced points from [0,1] \n",
    "x_train = torch.linspace(0, 1, 15)\n",
    "\n",
    "# The true function is sin(2*pi*x) with Gaussian noise N(0, 0.04)\n",
    "y_train = torch.sin(x_train * (2 * math.pi)) + torch.randn(x_train.size()) * math.sqrt(0.04)\n",
    "\n",
    "# Plot training data as black stars\n",
    "plt.plot(x_train.numpy(), y_train.numpy(), 'k*')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "checked-municipality",
   "metadata": {},
   "source": [
    "## Building the model\n",
    "GPyTorch offers a flexible way for us to build GP models, by constructing the components of the model by ourselves. This is analogous to building neural networks in the standard PyTorch library. For most GP regression models, you will need to construct the following components:\n",
    "1. **A GP Model**: For exact (i.e. non-variational) GP models we will use `gpytorch.models.ExactGP`\n",
    "2. **A likelihood function**: The likelihood function for GP regression, we commonly use `gpytorch.likelihoods.GaussianLikelihood`.\n",
    "3. **A mean function**: The prior mean of the GP. If you don't know which mean to use, `gpytorch.means.ConstantMean()` is usually a good place to start.\n",
    "4. **A kernel function**: The prior covariance of the GP. We'll use the [Spectral Mixture (SM) kernel](https://arxiv.org/pdf/1302.4245.pdf) (`gpytorch.kernels.SpectralMixtureKernel()`) for this tutorial.\n",
    "5. **A multivariate normal distribution**: An object to represent the multivariate normal distribution (`gpytorch.distributions.MultivariateNormal`)\n",
    "\n",
    "Now, let's build our GP model by constructing its components:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "roman-belgium",
   "metadata": {},
   "outputs": [],
   "source": [
    "# A GP model with constant mean function and spectral mixture (SM) kernel\n",
    "class SpectralMixtureGP(gpytorch.models.ExactGP):\n",
    "    def __init__(self, x_train, y_train, likelihood):\n",
    "        super(SpectralMixtureGP, self).__init__(x_train, y_train, likelihood)\n",
    "        self.mean = gpytorch.means.ConstantMean() # Construct the mean function\n",
    "        self.cov = gpytorch.kernels.SpectralMixtureKernel(num_mixtures=4) # Construct the kernel function\n",
    "        self.cov.initialize_from_data(x_train, y_train) # Initialize the hyperparameters from data\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # Evaluate the mean and kernel function at x\n",
    "        mean_x = self.mean(x)\n",
    "        cov_x = self.cov(x)\n",
    "        # Return the multivariate normal distribution using the evaluated mean and kernel function\n",
    "        return gpytorch.distributions.MultivariateNormal(mean_x, cov_x) \n",
    "        \n",
    "# Initialize the likelihood and model\n",
    "likelihood = gpytorch.likelihoods.GaussianLikelihood()\n",
    "model = SpectralMixtureGP(x_train, y_train, likelihood)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bound-grass",
   "metadata": {},
   "source": [
    "## Training the model\n",
    "Now that we have built the model, we can train the model to find the optimal hyperparameters. Training a GP model in GPyTorch is also analogous to training a neural network in the standard PyTorch library. The training loop mainly consists of the following steps:\n",
    "1. Setting all the parameter gradients to zero\n",
    "2. Calling the model and computing the loss\n",
    "3. Calling backward on the loss to fill in gradients\n",
    "4. Taking a step on the optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "living-daniel",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iter 1/50 - Loss: 1.276\n",
      "Iter 2/50 - Loss: 1.243\n",
      "Iter 3/50 - Loss: 1.214\n",
      "Iter 4/50 - Loss: 1.183\n",
      "Iter 5/50 - Loss: 1.151\n",
      "Iter 6/50 - Loss: 1.123\n",
      "Iter 7/50 - Loss: 1.095\n",
      "Iter 8/50 - Loss: 1.069\n",
      "Iter 9/50 - Loss: 1.043\n",
      "Iter 10/50 - Loss: 1.019\n",
      "Iter 11/50 - Loss: 0.997\n",
      "Iter 12/50 - Loss: 0.977\n",
      "Iter 13/50 - Loss: 0.955\n",
      "Iter 14/50 - Loss: 0.934\n",
      "Iter 15/50 - Loss: 0.912\n",
      "Iter 16/50 - Loss: 0.889\n",
      "Iter 17/50 - Loss: 0.862\n",
      "Iter 18/50 - Loss: 0.832\n",
      "Iter 19/50 - Loss: 0.799\n",
      "Iter 20/50 - Loss: 0.763\n",
      "Iter 21/50 - Loss: 0.725\n",
      "Iter 22/50 - Loss: 0.685\n",
      "Iter 23/50 - Loss: 0.645\n",
      "Iter 24/50 - Loss: 0.605\n",
      "Iter 25/50 - Loss: 0.565\n",
      "Iter 26/50 - Loss: 0.528\n",
      "Iter 27/50 - Loss: 0.493\n",
      "Iter 28/50 - Loss: 0.462\n",
      "Iter 29/50 - Loss: 0.435\n",
      "Iter 30/50 - Loss: 0.411\n",
      "Iter 31/50 - Loss: 0.392\n",
      "Iter 32/50 - Loss: 0.377\n",
      "Iter 33/50 - Loss: 0.362\n",
      "Iter 34/50 - Loss: 0.347\n",
      "Iter 35/50 - Loss: 0.332\n",
      "Iter 36/50 - Loss: 0.317\n",
      "Iter 37/50 - Loss: 0.305\n",
      "Iter 38/50 - Loss: 0.296\n",
      "Iter 39/50 - Loss: 0.290\n",
      "Iter 40/50 - Loss: 0.286\n",
      "Iter 41/50 - Loss: 0.280\n",
      "Iter 42/50 - Loss: 0.274\n",
      "Iter 43/50 - Loss: 0.264\n",
      "Iter 44/50 - Loss: 0.250\n",
      "Iter 45/50 - Loss: 0.233\n",
      "Iter 46/50 - Loss: 0.213\n",
      "Iter 47/50 - Loss: 0.191\n",
      "Iter 48/50 - Loss: 0.170\n",
      "Iter 49/50 - Loss: 0.151\n",
      "Iter 50/50 - Loss: 0.129\n"
     ]
    }
   ],
   "source": [
    "# Put the model into training mode\n",
    "model.train()\n",
    "likelihood.train()\n",
    "\n",
    "# Use the Adam optimizer, with learning rate set to 0.1\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.1)\n",
    "\n",
    "# Use the negative marginal log-likelihood as the loss function\n",
    "mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)\n",
    "\n",
    "# Set the number of training iterations\n",
    "n_iter = 50\n",
    "\n",
    "for i in range(n_iter):\n",
    "    # Set the gradients from previous iteration to zero\n",
    "    optimizer.zero_grad()\n",
    "    # Output from model\n",
    "    output = model(x_train)\n",
    "    # Compute loss and backprop gradients\n",
    "    loss = -mll(output, y_train)\n",
    "    loss.backward()\n",
    "    print('Iter %d/%d - Loss: %.3f' % (i + 1, n_iter, loss.item()))\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "solved-belly",
   "metadata": {},
   "source": [
    "## Making predictions with the model\n",
    "Next, we can make predicitions with the model by putting the model and likelihood in eval mode. We will evaluate the model using 50 equally-spaced points from $[0, 5]$ as our test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "metropolitan-potato",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeIAAAFpCAYAAABAniu4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABc+klEQVR4nO3dd3ib1dk/8O/RXpa85O0kdoaTeGU4m5ABYYSwVwItb0OBQksLhULLC7yBjvfXF1rogNLSlhZKCHs1rCRlZO/p7MR2vPfQ3uf3h2wnsqXEssYjyffnunxFlh49z7Fi69Y55z73YZxzEEIIIUQYIqEbQAghhIxkFIgJIYQQAVEgJoQQQgREgZgQQggREAViQgghREAUiAkhhBABhRyIGWMKxthOxtgBxthhxtjT4WgYIYQQMhKwUNcRM8YYADXn3MQYkwLYDOABzvn2cDSQEEIISWSSUE/AvZHc1PuttPeLqoQQQgghQxCWOWLGmJgxth9AK4D1nPMd4TgvIYQQkuhC7hEDAOfcDWAKYywZwAeMsRLOeeW5xzDG7gFwDwCo1erpEydODMelCSGEkJi3Z8+eds653t9jIc8RDzohY/8DwMI5/02gYyoqKvju3bvDel1CCCEkVjHG9nDOK/w9Fo6saX1vTxiMMSWAJQCOhXpeQgghZCQIx9B0NoBXGWNieAP725zztWE4LyGEEJLwwpE1fRDA1DC0hRBCCBlxwpKsRQghJLycTifq6+ths9mEbgoJgkKhQF5eHqRS6ZCfQ4GYEEJiUH19PZKSkjBmzBh46yaRWMc5R0dHB+rr61FQUDDk51GtaUIIiUE2mw1paWkUhOMIYwxpaWlBj2JQICaEkBhFQTj+DOf/jAIxIYQQv+rr63Httddi/PjxGDt2LB544AE4HA4AwD//+U/cf//9ArdwMI1G4/d+sViMKVOmoLi4GOXl5fjtb38Lj8dz3nPV1NTgjTfeiEQzfVAgJoSQBNHU1IQFCxagubk55HNxznHDDTfguuuuw8mTJ3HixAmYTCY8/vjjYWipfy6XK2LnViqV2L9/Pw4fPoz169fjs88+w9NPn3+zQArEhBBCgvKLX/wCmzdvxs9//vOQz/Xll19CoVBg5cqVALw9yueffx6vvPIKLBYLAKCurg4LFy7E+PHj+4Oa2WzGVVddhfLycpSUlOCtt94CAOzZswcLFizA9OnTcfnll6OpqQkAsHDhQjz44IOoqKjAr371K4wePbq/p2o2m5Gfnw+n04nTp0/jiiuuwPTp0zF//nwcO+atG1VdXY05c+agtLQUTzzxxJB+toyMDLz88st44YUXwDlHTU0N5s+fj2nTpmHatGnYunUrAOBnP/sZNm3ahClTpuD5558PeFzIOOdR/5o+fTonhBAS2JEjR4Z8rEKh4PDueufzpVAohn393//+9/zBBx8cdP+UKVP4gQMH+D/+8Q+elZXF29vbucVi4cXFxXzXrl383Xff5XfddVf/8d3d3dzhcPA5c+bw1tZWzjnnb775Jl+5ciXnnPMFCxbw++67r//4a665hn/55Zf9x333u9/lnHO+ePFifuLECc4559u3b+eLFi3inHN+9dVX81dffZVzzvkLL7zA1Wq135/H3/06nY43Nzdzs9nMrVYr55zzEydO8L4Y9dVXX/Grrrqq//hAxw3k7/8OwG4eICZSj5gQQuJcVVUVbrvtNqhUKgCASqXC7bffjurq6ohed8mSJUhLS4NSqcQNN9yAzZs3o7S0FOvXr8dPf/pTbNq0CTqdDsePH0dlZSWWLFmCKVOm4Je//CXq6+v7z3Prrbf63O7rRb/55pu49dZbYTKZsHXrVtx8882YMmUKvve97/X3qLds2YIVK1YAAL797W8P6+dwOp24++67UVpaiptvvhlHjhwJ6bhg0TpiQgiJc9nZ2dBqtbDZbFAoFLDZbNBqtcjKyhr2OSdPnox3333X5z6DwYDa2lqMGzcOe/fuHZQhzBjDhAkTsHfvXnz66ad44okncMkll+D6669HcXExtm3b5vdaarW6//Y111yD//7v/0ZnZyf27NmDxYsXw2w2Izk5Gfv37/f7/OFkKldVVUEsFiMjIwNPP/00MjMzceDAAXg8HigUCr/Pef7554d0XLCoR0wIIQmgpaUF9957L7Zv345777035IStSy65BBaLBa+99hoAwO124+GHH8Z3vvOd/p73+vXr0dnZCavVig8//BDz5s1DY2MjVCoVvvWtb+GRRx7B3r17UVRUhLa2tv5A7HQ6cfjwYb/X1Wg0mDFjBh544AEsW7YMYrEYWq0WBQUFeOeddwB4p1QPHDgAAJg3bx7efPNNAMDq1auH9LO1tbXh3nvvxf333w/GGHp6epCdnQ2RSIR//etfcLvdAICkpCQYjcb+5wU6LmSBxqwj+UVzxIQQcn7BzBFHSm1tLV+2bBkfN24cLyws5Pfffz+32Wycc87/8Y9/8GuvvZYvXLiQjxs3jj/11FOcc84///xzXlpaysvLy3lFRQXftWsX55zzffv28fnz5/OysjI+efJk/vLLL3POvXPEfcf0eeeddzgA/vXXX/ffV1VVxS+//HJeVlbGJ02axJ9++un++2fPns1LSkr4448/HnCOWCQS8fLycj558mReVlbGn332We52uznn3vne0tJSXlZWxh999NH+czgcDr5o0SJeVlbGn3vuuYDHDRTsHHHY9yMeCtqPmBBCzu/o0aOYNGmS0M0gw+Dv/y6i+xETQgghZPgoEBNCCCECokBMCCGECIgCMSGEECIgCsSEEEKIgCgQE0IIIQKiQEwIIcQvxhi+9a1v9X/vcrmg1+uxbNkyAVuVeCgQE0II8UutVqOyshJWqxWAt5JWbm6uwK1KPBSICSGEBLR06VJ88sknAIA1a9b0b7AAeLcpvPPOOzFz5kxMnToVH330EQAE3C7w66+/xsKFC3HTTTdh4sSJuP322yFEUalYQ5s+EEJIjHvwQSDAfgfDNmUK8LvfXfi45cuX4+c//zmWLVuGgwcP4s4778SmTZsAAL/61a+wePFivPLKK+ju7sbMmTNx6aWXIiMjA+vXr4dCocDJkyexYsUK9FVT3LdvHw4fPoycnBzMmzcPW7ZswUUXXRTeHy7OUCAmhBASUFlZGWpqarBmzRosXbrU57F169bh448/xm9+8xsAgM1mQ21tLXJycnD//fdj//79EIvFOHHiRP9zZs6ciby8PADAlClTUFNTQ4FY6AYQQgg5v6H0XCPpmmuuwU9+8hN8/fXX6Ojo6L+fc4733nsPRUVFPsc/9dRTAbcLlMvl/bfFYjFcLlfkf4AYR3PEhBBCzuvOO+/EqlWrUFpa6nP/5Zdfjj/+8Y/987z79u0DEMHtAhMUBWJCCCHnlZeXhx/96EeD7n/yySfhdDpRVlaG4uJiPPnkkwCA73//+3j11VdRXl6OY8eOQa1WR7vJcYW2QSSEkBhE2yDGL9oGkRBCCIkjFIgJIYQQAVEgJoQQQgREgZgQQggREAViQgghREAUiAkhhBABUSAmhBASUHNzM5YvX46xY8di+vTpWLp0qU/JyqHatGkTiouLMWXKFDQ0NOCmm27ye9zChQsx0pa3UolLQgiJA8+vDz74nc+Pl0y44DGcc1x//fX4r//6L7z55psAgAMHDqClpQUTJlz4+edavXo1Hnvssf79jd99993gG52gqEdMCCHEr6+++gpSqRT33ntv/33l5eW46KKL8Mgjj6CkpASlpaV46623AATe5vBvf/sb3n77bTz55JO4/fbbUVNTg5KSEgCA1WrF8uXLMWnSJFx//fX9ex8D3k0l5syZg2nTpuHmm2+GyWQCAIwZMwarVq3CtGnTUFpaimPHjgEATCYTVq5cidLSUpSVleG9994773liBQViQgghflVWVmL69OmD7n///fexf/9+HDhwABs2bMAjjzyCpqYmAN5607/73e9w5MgRVFVVYcuWLbjrrrtwzTXX4Nlnn8Xq1at9zvXSSy9BpVLh6NGjePrpp7Fnzx4AQHt7O375y19iw4YN2Lt3LyoqKvDcc8/1Py89PR179+7Ffffd17/70y9+8QvodDocOnQIBw8exOLFiy94nlhAQ9OEEEKCsnnzZqxYsQJisRiZmZlYsGABdu3aBa1WG/Q2hxs3buyvY11WVoaysjIAwPbt23HkyBHMmzcPAOBwODBnzpz+591www0AgOnTp+P9998HAGzYsKF/CB0AUlJSsHbt2vOeJxZQICaEEOJXcXFx0HO54drmkHOOJUuWYM2aNee9zoWucaHzxAIamiaEEOLX4sWLYbfb8fLLL/ffd/DgQSQnJ+Ott96C2+1GW1sbNm7ciJkzZw7rGhdffDHeeOMNAN6h8IMHDwIAZs+ejS1btuDUqVMAALPZfMFs7SVLluDFF1/s/76rq2tY54k2CsSEEEL8Yozhgw8+wIYNGzB27FgUFxfjsccew2233YaysjKUl5dj8eLFeOaZZ5CVlTWsa9x3330wmUyYNGkS/ud//qd/Tlqv1+Of//wnVqxYgbKyMsyZM6c/KSuQJ554Al1dXSgpKUF5eTm++uqrYZ0n2mgbREIIiUG0DWL8om0QCSGEkDhCgZgQQggREAViQgghREAhB2LGWD5j7CvG2BHG2GHG2APhaBghhIx0QuTwkNAM5/8sHD1iF4CHOeeTAcwG8APG2OQwnJcQQkYshUKBjo4OCsZxhHOOjo4OKBSKoJ4XckEPznkTgKbe20bG2FEAuQCOhHpuQggZqfLy8lBfX4+2tjahm0KCoFAo+iuLDVVYK2sxxsYAmApgh5/H7gFwDwCMGjUqnJclhJCEI5VKUVBQIHQzSBSELVmLMaYB8B6ABznnhoGPc85f5pxXcM4r9Hp9uC5LCCGExLWwBGLGmBTeILyac/5+OM5JCCGEjAThyJpmAP4O4CjnPLb2liKEEEJiXDh6xPMAfBvAYsbY/t6vpWE4LyGEEJLwwpE1vRkAC0NbCCGEkBGHKmsRQgghAqJATAghhAiIAjEhhBAiIArEhBBCiIAoEBNCCCECokBMCCGECIgCMSGEECIgCsSEEEKIgCgQE0IIIQKiQEwIIYQIiAIxIYQQIiAKxIQQQoiAKBATQgghAqJATEiMsrvcQjeBEBIFIW+DSAgJv1ajDe/srodOKcWEzCRMyNQgWSUTulmEkAigQExIjDHanPh4fyMcLg/ajHa0Ge3YcqodmVoFJmRqMD4zCTqlVOhmEkLChAIxITHE4fLgo/2NMNpcgx5rMdjQYrBh08l2ZOsUmJStRXl+cvQbSQgJKwrEhMQIj4fj00NNaDPaL3hsU48NTT02ONwezBiTGoXWxaceqxON3VaY7C6ka+TISJJDLae3PRJb6DeSkBjx9YlWVLebg3rOllPtSFZKMT4zKUKtih+cc7SZ7GjstqGx24rGbqvfkQWNXIIMrRwZSQpkaOXI1CqgoeBMBES/fYTEgD1nOnGgrifo53EOfHG4GUkKKbJ0igi0LPbVdVqw+0wnmnpssDs9FzzeZHfB1OZCVdvZDz1JCgmKspJQnKNDqpqS4kh0USAmYdXYbYVcIkKaRi50U+LGqVYjNp1sH/bznW6Ojw80YPnMUdAqRk4Sl9vDse10B3af6QTnoZ3LaHNhd00Xdtd0IVunQHGODhOyNJBLxOFpLCHnQYGYhA3nHBuOtqDD5EBuihLleckYl6GBWMSEblrMauqx4vPK5pADidnuxkf7G3FLRd6ICB7dFgc+q2xGc48t7Ofum3//5kQrxmVoMDlbh/xUJRhL/N9jj4fD5fH+MnLw/t9Lzr3fA4CIMcjEIojo7zpsKBCTsKlqN6PD5AAANHRZ0dBlhUomRnGODqW5OuhUI6e3NhQ9Fu8yJac7xCjcq91ox6eHmnBteW5Cv0kebuzB18fb4HBdeBg6FE43x9EmI442GfuHridkJiFTG59TADanG10WB0w2F0x2FywOd++/LpjtbpjtLlid7iF/KJSKGWQSEaRiEWQSEWS9/8olYsilIigkYiikIiik4t6vvvvEkElE9AH9HBSISdjsqekadJ/F4caumk7sPtOJ0WkqlOclo1CvEaB1sYVzjo8PNsLiCG/1rJp2C74+0YrFEzPDet5YYHO68dWxVhxrNkb92ucOXeuUUozP1MRsUHa5Peg0O9BucqDDbEeHyYF2k91v4loonG4Op9sNYHi/wxKRN5CfG8zlvQFdIhZBLPL2vsUiBjFjEIm8t0WMQcSAvs8LZz84nNODh3fqwsM5PB7Azftu8977vY+7z7mv/3bvv7fOGAWZJDrFJykQk7Bo7Laiodsa8HHOvUGipt2C22ePQkZS7L2BRVNNhwXtQ1imNBwH6nqQrJJh2qiUiJxfCI3dVnxW2QyD1Sl0U9BjdfoE5b7KZxkCBGWT3dVf9KXNaEe7yY5uixOeUOc6osDl4XA53GH/MBou0XwNKRADON5sRIfJjhkFqZCKqfz2cOw+M7g3HMj+2m5cVpwVwdbEvoP13RE9/8YTbUhWShNi9GFvbRc2nWiPyeDSY3ViV00ndtV0QikTQ6+RIz1JDr1GDn2SHKlqWViGYC0OF0w2FzotDp/AG6tBjASHAjG8f+jNPTYcbTZiwYR0jMugNZnB6DQ7UNVmGvLxx5uNmD9eD6Us8ZOK/OmxOoNeLxwszoHPKptx8/Q8QXpq4bLtdAe2V3UI3YwhsTrcqO20oLbT0n+fWMSQqpb5BGUx6x1ePWfotW+41eJww2BzwmRzwWhzwWhzwmR3hS2PgMSmER+IWw22/sxLg9WJfx9owui0HiwsyqD1hEO050xXUFm/Lg/HwfpuzCpMi1yjYtih+p6Qs6SHwuHy4MP9Dbh1xqi4rE39zYk27A1ipCUWuT28v/dKSCAjfhz2QP3gIgpnOix4ffsZbDoZ+czMeGe2u3CsyRD08w7W98DtGXmf8t0ejsONwRfuGC6z3Y0P9zXAGkdDmJxzbDjSEvdBmJChGtGB2OZ043iz/yDi9nDsrunCq1trcCzAMQTYX9fdv+4wGCa7Cydbo5/9KrQTLcaoz+t1mh34+EADXO7Y/1Dp8XB8XtmMQw3R+7BCiNBGdCA+0mS44NyLye7CZ4eaz5sRPFI5XB4cCCHpaH/t8J8bryKdpBVIY7cNn1U2g8dgwlMfl9uDtYeaBFmeRIiQRnQgPuRnWDqQ6rbIJtfEo0MNPUOq7RuIt4LRyPmA02q0obE7/JWghupUqwlfH28T7Prn07f94+nWoSf9EZIoRmwgru2woNPsGPLxVe30BnEut4djX23oc3gjqVd8cBibOoTb/rpu7K7pFLoZPmxO7zz2udnGhIwkIzYQH2zoDur4DpMDPRbhiwnEiuPNxrBU6jnZaoLJHt6KP7HI7nLjeEtsDLluPtUeM3kP3RYH3tlTT1M/ZEQbkYHYZHfhdGvwQ83UKz5rTxh6w4C3Z32wrjss54plRxoNMZOBzzmw7nAL6gTugZ5qNeGNnbURqzBGSLwYkYH4UH3PsKr0RLoIQ7yobjeH9c3zYENPXGT0huJgEPkI0eD2cPz7YGNEdi+6EI+HY9PJNqw92BhSjgEhiWLEBWJPCOs467ussLviZz1mpIR7jtHqcCd0pmxdZ3D5CNFid3rw9u46bD3dHrU13Wa7C+/trcfumuCKwBCSyEZcID7dZhr23Kbbw1HbMbITSloMNtR3hX8+b18CD0+HssQr0twejh1VnXhjZy1aDJHtHTd0W/HGjtqI/P4QEs9GXCAOdYiwaoQPT++LUJZzu9Eu+JxlJAw3HyHa2o12vLmzDltOtUdkmmDPmU68u7t+RCTmERKsEVVrutPsQF1XaG/2Ne1mcM7B2Mjc1Lq2M3JBZV9dN/JTVRE7vxCGm48gBA/n2FndidNtJlw2OQtZutA3i2gz2rG9qgOnaH0wIQGNqEB8sL475Hkpi8ONph4bcpKV4WlUHOkyO2C2R26OvKrNhB6LEzpV/G1Q4I/Hw1EZh6UaO0wOvLWrDtNGJ2NOYRokQW4N2ml24HizESdbjegwxd7cOCGxZsQEYqfbgyPD2JzAn+p284gMxJGe2+Mc2F/fjQUT9BG9TrScbovfNdIe7q21frzZiAytAikqKVJUMqSoZUhRSaGS+b51dFscONFiwvEWIy1HIiRIIyYQH282BlwqwTkQzEhzVbsZ88alh6ll8aOhO/JzuIcbezCnMA0ySfynL/jb2SveePfEHTysrJCKkaKSIlklQ6fZEfFEL0ISWfy/2w1RoCQttwt45u7ReOnRPLTWDW1ItN1oh8E28qpsRSPb1e704ESMVKAKRafZkZDJZ31sTu8UzdEmAwVhQkI0IgJxc48t4JtF5VYNWmrlqD6iwLP3jsbnr6XB6bhw97hqhG0C0WNxhqWk5VAcaYyN8ouhiOaew4SQ+BaWQMwYe4Ux1soYqwzH+cLt6Hnmhrd+okNKphOPv1qN8otMWPd6Gn7zvdE4ue/8c8DVI6zcZX0UhqX7NHRb0RWDBTCGyuPh5/2dI4SQc4WrR/xPAFeE6VxhF6g0ZVuDFCf3qTH7yh7o0tz41mPN+N7/1sPDgZd+mo83nsmCqVvs97n1ndaYqR0cDQ1RLsIQrsQ6IVS1myOaXU4ISSxhCcSc840AYmtvtV4dJjt6rP7nc7d9ooNIzDHr8rPDiEUVFjzylzO4dEUH9n2dhP/33THY/pkWngEx1+XhI2rbtmhXQzraZIjpTezPh4alCSHBiNocMWPsHsbYbsbY7ra26G1OXtPhvzfsdDDsWqdDyRwTtGm+vReZnGPpyg48/NIZZI+24+3ns/DG/2UNOkdV28gYnjbanAE/zETumi6cicNyoia7CzXt8dduQohwohaIOecvc84rOOcVen301olWB3hTPLhZA7NBjLnLvL0XQ0crXnj4WzB0nv2QkDXage//ph5zl3Vj3zdJMBt8X66aDnPc9tqCIdResfE4PH20yRA3lbQIIbEhobOm7S43GgMEkW1rdUjPcWDcFG+gXrf6T6iu3I11r7/oc5xIBMy4zADuYTi6U+3zmNnuRosh8YsX1HcKE4hPt5pgc8bXXOvhOKykRQgRVkIX9KjtsPjd3q35jAxVlSosu6sNP7umDC7H2WC6de0abF27BhKZHM+sPQgAyJ9gQ1KqC0d2aFBxqe8a16p2U1hq8sYyoXrELg/H8WYjyvOTBbl+sOq7LOiyjLz15YSQ0IRr+dIaANsAFDHG6hlj3w3HeUMVKFt62yc6iKUezLzMgCde3YBpi5ZBKvcGU6lcgWmLr8YTr/2nf7ja1N2GyTPNOLZLBdeA99lEX09strsE3Uv3cBytKa5siJ+2EkJiR7iypldwzrM551LOeR7n/O/hOG+IbfKb7OOwMexar0X5RSZokt3QpmVAodLA5bBDIpPD5bBDodJAm6r3Ga4unm2CzSJGVaXv+uI2ox3GBK6yFUxv2OMGtn+mRVtD+DZtaDHY0G6K/eF/u8uNU63xXxGMEBJ9CTs03Wa0+y24v39jEmxmMeZcdXYuz9jdgbnLVmD20lux/dO3sHXtm9i6dk3/497h6o8AdODIdg0mTPUNTtXtZpTlJUfqRxFUMOuHN3+cjA9fyoBYwnHRtV247PZOKDWhr7U+0mjAxTG+EcTxZiOcbkrSIoQEL2GTtaoCDEtv+lAFqbwK6bm1/fetXPUCbvzhKuSOnYgbf7gKq974xs9w9SUYP9WCwzvUg7ZSDDQEngjqh9gj7myW4NNX0jFhmhkVSwzY+H4K/nflGGz5tw7uEPOtjjUb4PEz1x9L4mkInRASWxI2ENf4CY4Np+VoOKWF0/57rF/9op9neQUari67yImORhla62Q+x9d2WOB0J16VLavDjY4hDAtzDrzz+0wwEXDrQy249ccteOhPtcga48B7f8zEb+8djWO7VcNuh9nuRnWA9eCxoN1kR3MPbXxACBmehAzEVocbzQM2eXh0WRl+e9+nAKwAXsPWtWvw0GVFeHRZmd9z9A1XP/D7tzF32QoYu9oxeZY3GBze7ruMKVGrbDV0Wwf1/v3ZvSEJx/eosfTOdqRkeKcDcsfa8f1n6rFyVQOcToaX/zsPf30iBy21w5s/juUeZyUtWSKEhCAh54i9hTZ873vkz1/h/+4uB+fvgHu6IZUrUDpvCa6556d+z7Fy1Qv9t2/84areWy7kjrXh8HY1Ft/S5XN8dZsZY/WacP4YghtKopaxS4wP/5yBgmIr5l3d7fMYY0DpPDMmzbBg00fJWL86Fc/eMwY33N/aX0hlqGrazbA4XIM2pBea28NxrJmStAghw5eQPWJ/w9KnDhTC41aBe14alB0djMmzzag5ovRbZSvR1HdduJf/wZ8y4LAx3PLjZogC/DZJZByLbu7CY/+oQUGJFZ/+Ix0O24W3mjxXrAa8020mWB3xVXSEEBJbEi4QezwcNQOWLXHu3e5QoT6NucsKfYabg1U82+S3ypbR5kJrAm2Qbne50WY8//xw5TY19n+ThMtu60TmqAsv4UpKcePK/+qAxSjG7g3aoNsUi8PTtMEDISRUsTXOFwZNBtugsoi1xxVoOKXATT/SYO4y7zDz2eHm4OSNtwesslXdbkaGNjGqbDV22847P2w1i/DuHzKRXWDHoluGvvFWQYkVeeNs2PhBMmYv7QnYi/an3WhHq8EWM6+xweaMy40pCCGxJeF6xP6Gpbd9ooNM4cG0Rf6HNjO1CmQPsUylSISAVbYSaRnThYal1/4tHcYuMW59qBmSIPKvGAMuvqELrXVynNgbfCZ1LPWKjzQahpTMRggh55NwgXhgMLSaRNj3dRKmLTZAofa/xGj66BTcOiMfF0/QQyq+8NxloCpbzQYbLI7BRUTi0fkKeZw6qMS2T5Jx8fVdGFUUfNWrKQuMSEp1YeP7KUE/93iL0W/98GjjnMfUhwJCSPxKqEBstDkHzWse3q6G0y7CrMv9v2kqZWKMy9CAMYbpo1Pw7dljkJ96/p7a+KkWSKQeHN7umyXNeWL0ih0uT8BdpRx2hneez0RqlgNX3NExrPNLpMBFV3fj2G41WmplF37COawON07HwD7QdZ1WGKK8RzMhJDElVCD2tyH70Z1qJKW4kF/kP5FqcrYWYtHZXrBOJcVN0/Nw6aRMyKX+Xx65kmP8VAuObE/MKltNPdaAe+quez0NbQ0y3PJgK+TK4fdM51zVA4nUg40fJAf93J3VnYLvA729angfQgghZKCECsQDqy+53cCx3WpMmmkOmBRUmqvzf3+eDnfMGYNCvdrv48WzzehoGlxl60yArRfjSaBh6cYqGb5+JwUzL+/BhGmhJSlpkt2YfqkRuzdoBy0Fu5A2o13QYeETLUbBtoYkhCSehAnEbg9H3YDqVjWHlbCaxJg0038vNS9FiRR14KFRjVyCa6fkYmlp9qC540BVthwuT1AbJcSiQPWlt3+mg1jKcfXdbWG5zsXXdcFpF2HbJ/4/DJ3PttMdcLiiX1bU5fZg88ngl70RQkggCROIG7qsg96Yj+xUQyzhKArQeysJ0BseqCgrCaPTfANust6F3HG2QYEYAKrahZ/DHC6X24MWP3WTOQcqt2pQNM0CtTY8ATC7wIEJU83Y/HEK3EHmuJnsLuw503XhA8NsX103emhumBASRgkTiP1tCnB0hxqFpRa/2dIKqRjjM4ZekrIgfXDAnTwrQJWtOJ4nbuqxweVnaL3hlBzdbVKUzAvvh4yLb+yGoUOCA5uSgn7unjOdfre6jBSLw4Wd1UNfM00IIUOROIF4QCZtZ7MEzWfkmBxgWHpSdhIk4qH/+IV6NdiAlU2Bqmx1WZzoMjuGfO5YEmju89BWDZiIY/Ks8AbiiRVm6PMc+Ob9lKDX5DrdHFtORW+YeOspYYbDCSGJLSECcbfFgS6L73Dhkd7gOGmW/0AcKEkrEJVMgqwBFZ3OrbI1UKD9kGNdfYD57cqtGhSWWKHRhTcQiUTeueK64wrUHAm+YtbRJgNajZEvLSp0ghghJHElRCD2t2ToyA4N9LkOZOQNns/LTVYiTSMP+jqFA3ZXEom8w9OJUmXL4+Fo7hkciNsbpWiqlqNkbmTmviuWGKDUuLHxg+ALfHAObDwR+V7xxhNtAZd0EUJIKBIiEA/cC9huZTi1XxkwW3qoSVoD+ZsnDlRlq7HbCrsrvnblaTXa4XQPDjaVW70/d8mcyARiuZJj9tIeHNysQWdL8OXP6zotES3yUdVmSsj9pgkhsSEhArF9wLzdqf0quJyi/iVG55JLRRifObx9g/VJcmiVvoWVJ0y1QCIbXGXL7eFxtyFAo5/eMOAdls4ptCEtO3KJURdd0w0GYMvHycN6/uaT7fBEYP22x8OxiZYrEUIiKCEC8UBHdqohV3pQWDo4EE7K0kIaRJLWQIXpahg6WvHCw9+CobMNMgXH+Cn+q2xVtcXX8HRT9+C5VmOXGNVHlCiZG9mfJSXDhbL5Jmz7VAe7Nbi9igGg0+zAwYbwb0l4oL4bnVFOvHO7gE//kYYnbxqLf/1vFk4fVNLmEoQksIQLxJx7ly1NmGb2uyvQcIel+xTq1Vi3+k+ortyNda+/COBsla2BdZPPdJgFL8UYjCY/PeIjO9TgHobSMC9b8ufiG7pgM4uxa33wexUD3rKTA7fADIXN6cb2quguV+psluDFn+Rjw5o05I614eguNV78ST6evWc0Nn+sg82ccH+yhIx4ibcfcbUM3e1SXO5nQ4JsnQL6pOCTtPoolUrYbGd7jVvXrsHWtWsglhYCOI0jO9TIGn2292RxuNFssCFbp/RztthisDlhtA0eeq7cqkFKphM5hcHvshSsMZNsGDXRik0fpmDe1T2DlotdiNXhxq6aTswfrw9Le8Id2C9k/zcavP27TIAD3/7vRkxdaILDxrDv6yRs+Xcy3n8hE2v/psf0SwyYu6wbuWPjc4kcIcRXwn287ltK5C9RK9TecFVVFW677TbIFd7AKpUrMG3x1XjyX68FrLJVHSfD0/6Gpe1WhuN7VCiZawo6KA7XnKU9aKuXof7k8D4w7a8NT+WrLrMDB+vDP9Ttj93K8NZzmXjtVznIHOXAwy+dwdSF3hEImYJj1hUGPPRiLR784xmUX2zErvVa/Pa+MfjDg/loOB3c7lWEkNiTgIFYjbzxNmhTfXsyMokIRVnBV286V3Z2NrRaLRx2OyQyOVwOOxQqDbSp+oBVtuJlPbG/RK3je9RwOUUojdCyJX9K5pogEvNhVdoCAJeH4+vjrXC5h7/e2eJwYcPRlqhs3tFwWo7nfzAaO7/Q4tIVHbj/t3UBk+JGFdmx4ictWPVGFa65pxWdzVL89fE89HSII95OQkjkJFQgNvWIcOaYApNnmX0SqgBvJa1QkrT6tLS04O577sGDf3gbc5etgLHLm1EbqMpWm9EOoy32axP76xFXbtVAleRGQUn0NrFQaz2YMNWC/d9ohp2gVNVmxmvbzgS9pIlzjoP13Xh165mAhU3ChXNg4wfJ+N2P8mGziHDvr+uxdGUHxEOYLFJrPVh4Uzfu+X/1sFtFeOWpXDjsURqyIISEXUIF4mO7vIlFk2aaByVUhTos3ef999/HX/78EmZMm4obf7gKK1e9AOD8VbZivbiHw+VBm9F3DtjtAg7vUKN4tgniKHe4yi82orN5+MPTANBjdeLj/Y34YF/9kMqNthpseHNXHf5ztDUq88KfvJKOD1/KQNE0C37ylxqMnxp84M8pcOD2nzWh/oQcbz2XSZnVhMSphArE3t5oC37/ozHYunYNOOfYunYNHrqsCKMzgq/adD4FA/YpFomAyTPjs8pWi8E2qGpU1SElrEZxxKppnU/f8PT+jaFNJQBATbsF/9p+BptOtvmtE21zuvHVsVas2VmHZj+7TkVCa50UX7+bghlLevDdnzeGVDa0ZI4ZS1e2Y99XWvznzfD+jhNCoiNhArHbDRzbrcaUBcC0RVdBKvfWLZbKFbjyuptRXV0d1usV+qmyVVDcAJtFjMptvj2wuk4LnCHMWUZao5+NHg5t1UAq96BoevSLkvQNTx/YOPzh6XO5PRy7a7rw6tYaHG06Wy/6WLMBr22rwf667qiWr/z4ZT2kcg+W3dUeliS4xbd2YdpiAz79h76/ChohJH4kTCA+c0QJq0mM8vluKFQauBxnE6qy01ORlZUV1uulaeRIVvkuVK4+/CwAG9avrvW53+nmEZ9zDEXTgJ7guXsPyxTCjHdOWRD68PRAJrsLn1c24+1ddXhvTz0+O9QMsz26ZUiP71HhyA4NltzWiaSU8FybMeCWH7dgVJEVr/86G41VlElNSDxJmEB8eIcaIjHHhOkWGLs7MHfZCjzw+7ex6Prb0dXRGpFr9tWefnRZGR66rAg7Pv8ngC/RVD0BD11WhEeXlfUfW90e/SHeoeCcDwrEkdp7OBjFc3qHp78JfXh6oIZuqyC1o91u4KO/6JGW7cDF13WH9dwyOcedTzVCofbg76tyYeqmTGpC4kXCBOKjO9QoLLFCqfZg5aoXcOMPVyF37EQ89evn8P7770fkmmN7d2N64tUNmLZoWe9w+L8BjMPkWffiidf+039sdXts1p3uNDsGJSdFau/hYKi1HkyYZsGBjUkJk4S04zMdmmvkuPrudkhk4f+htGlufPfpBhi7xPjnz7MH5SoQQmJTQgTitiYxms/I/QYOfzsmhUtOshJyqQjatIz+4XCxdD0AwGyYD23q2QpPBqsTHabIV6cK1sDeMBC5vYeDNeViIzpbpKg7Eb7haaFYTSJ89moaxpZZIlouNH+CHSt+0oyqShXe+yNlUhMSDxIiEO/d7K10NXC3JYVUjJwIlpcUixjGpHkDfd9w+IN/eA5q7Rm0NZQOOr6mI/aypwcmaoWy97BaLsbNFXmYGGLhlD4lc00QSzgOhCF7Wmjr30iFxSDGtfe2RbxK2dSFJiy5rQM7Ptdh04fJkb0YISRkCRKIFUjLcUCf5zsWNyZNBZEosu96fT3uc4fD5y7TwGoqGVRlKxaHpwf2iCu3eofbg917WKuU4paKfOSlqHBlaTaWTM6ETBLar5cqyYMJ08zYH+fD020NUmz6MAUzLjMgb1x0RkUuv6MDJXNM+Pdf9ag/Ff8jCoQksrgPxBYLcHiPHJNnmgf1NAau9Y2EgnQ1RAMuPHm2GdzDcGyX7/Ubu62wu6KbpXs+VocbXRbfpVaHtqqD3ns4TSPDLRV5SFadzdYtydVhxcxRIW2yAQDlF5vQFefD0//+qx4SKcfS70RvX2ORCLj14WaodW6s/nXWiK68xbk3UY6QWBX3uy99+SXgtIswecAmDyJ2dtg4khRSMbKTFWg4Z3lS/gQbklJcOLxdjemXGPvvd3s46jotGJcRG0OtTT1Wn56msUuMmsNKLLl96Fv/ZekUuG5KLpSywVm6qWoZls/Ix6ZT7dhf2z2sNpbM8Q5P7/8mCaOKYm+O/UJO7leicqsGS1e2QZsW3Wig1nqw4ifN+MtjeVj7t3Tc8IO2qF4/2hx2hvYGKVrrZWjr/Wqtk6KtQQa7VYTsAjvyJ9iQP8H7b/YY+5BKihISaXH/a/jVV4Bc6cHYMt+5zpxkBRTS6CzhGKtX+wRikci7+9PBzRq4XfD5Y69uj6VA7DssfXi7GpwPfe/hUakqXF2ec94haIlYhEVFGRiVqsL6Iy2wOoILRn3D0wc2JeHqu8NTACNaPG7goz/rkZLpxIIbuwVpQ9F0Cy6+vgsbP0jBpBlmTJoZe9MjoehqleCLf6Xh5H4Vulsl4PzsL0hyuhP6PCemLjRCrvKg4ZQcBzYlYfunyQAAidSDnLF2jCqyIX+CDSVzzFBqYrfwDklccR+In30WGD2/Ge4By0EKozAs3X+tdA02nvAddiyebcLOL3SoqlRi/JSzQfpMDCVsDUzUOrJ96HsPj8vQ4MqSLEiGuJHGWL0GGbPk+LyyOejiJlMuNmHNbzSoPa7A6InRKUMZDjvXadFYpcAdjzdCGoHlSkN11XfbcWKfCm/+NguP/OUMNMnxP05rNYmw4c1UbPogGYA3sW/m5Q5k5Dqhz3dAn+uAXDn4Necc6GiSou64AnUn5ag9rsDOL3TY/FEKJDIPplxswuwre1BQYo2rD30kvsV9IBaJgKw8Nxq6fO8vSB+8+UKkpKhlSFFJ0WU5myw2YZoFEqkHR7ZrfAKx0eZCm9Ee8txpqDwejhaD7ZzvgdOVSpTNu/Dew8U5Wlw6KTPoRLgkhRQ3TsvDK1uqYbQNfQ76bPa0Jm4Csc0swqf/SEdBsRXlFwtbzEUq4/jWz5rw/A9H4e3nM7Hyqca4DTIuB8OWtTqsX50Gq0mE6ZcYcOV3OpCSMbTfJ8aA9Bwn0nOcmLrIO23kcQP1p+TY+YUOe75Mwu4NWmTk2zH7SgMqlvQIvoyPJL64T9byJ1klRao6umX+CvW+gV+u5BhXbu0d7vU9NhaWMbUa7XC6zzasuUYGq1GMwtLzD11OyU/GksnBB+E+IhHD1FHJQT1HqfGgaLo5rop7bHgzFaZuCa67rzUmgl5OoQPL7mxH5TYNtn8Wnp3IosnjAfZ9lYRf3zUaH/05A3njbPjxi7W47dGWIQfhQERi717PN/2oFU+tqcLyh5uhSvLg45f1eHrFWLz2q2yc2KuCh+IxiZC47xH7E8kiHoEU6tXYc8a3W14824T3XshEW70UGflne8vV7WbMGJMa7Sb6aOzxHR4+fVAFAIPm2s8lk4gwd1waWIiRpThHh+1VnX53QwqkfL4JR3ZoUHtMgdGTYrtXbOoRYdOHyZh+iQH5E2InwWz+9d04ulONj17SY1y5Bfrc+Ci9dfqgEh//VY+64wpkF9rxvf+tR1FFZOa65UqOmZcbMPNyA5qqZdj+mQ67/6PF/m+SkJrlwKzLDZhxmQHJ+tCCf6xyOQG3i3k/8HKG/s+9HP0fgsUSDpmCx8QHzESRkIG4MIrD0n1ydEoopGKfcpGTZ5vx3gvA4e0aZOSfDdJN3TbYnO6oJZP509TtG8yqDimRkulEambgN5jiHC3kktDbrJCKMTlHG1Qmdf/w9CZNzAfiLR8nw2kX4ZLlQ88+jwaRCFj+SAue/d5ovP7rbPzo+dqYzhr2eIB1r6dh3etpSE53YsVPmjH9EgNEUfqzyS5w4Prvt2HZXe04uEmDHZ/r8Nmr6fj8X2komm7B7Ct7MHmWCRLphc8lNKeD9WeQG7skMHWLz/nyfm/sFsNmHtqLy0QccqUHCqUHcrUHCtXZL7nKA1WSG6okN9Tas7dV59ymQO4rhv8Mh0cmESE3JXLVtAIRiRgK0lU42nR2uVJKhgvZhXYc3q7GopvPBmIP56jttGBCpnDZ003n9Ig5B04fUmLieXoZjHmHpcNlWn4KDtR1D3mo+dzh6VjOnnbYGTZ/nIzJs0zIGu248BOiLDndhVseaMGrv8zButVpuPK/OoRukl9Wswirf52FIzs0mHFZD268v1WwncCkMo7plxgx/RIj2hul2PmFFjvX6fDPn+dAo3OhYokBs64wIHOU8P/fVrMILbUytNbK0NL3VSdDZ5PUJ6OcMQ611g1Nsvcrd5zNe1vnhlTOe48BgHMCJvPe53Iy2K0i2Cwi2C0i2Mze2zaLCN3tElhNYlhNIjjtgWc+RSJ+NoCrPVCo3Ofc9kAm5xBLOSRSDrHE+yWRnv0SifnZXntvb73vvYRzBvSuHXc7GdxuBrer78vb43e7mfex3vv7RgLcLgaXg8HlYtg3leG1VyPy3zRIWAIxY+wKAL8HIAbwN875r8Nx3uEYnaaCOMLVtAIp1Gt8AjEAFM8y4cu3UmE2iKDWnh2KrW43CxaIDTanT7JU1SEjTN0S5BS2BHxOQbrap2BHqHQqKcbqNTjVOvREpvKLjTE/PL17vRbmHgkW3tx14YMFUn6xCTMu68GGNamYWGFGQXFsvZYttTK88lQOOpqkuOH+Fsy7uidmPnil5zixdGUHLr+jA8d3q7Hjcy02fpCCr99NRe44G8aWWlFYakVBsTVs21z643YD7Q1SNFbJ0VQtR2OV96u77Wz3XCz1QJ/rRN44O6YvNiIz34GMUQ5oU11Qa90RH1lw2BksRjEsRpH3X8PZ2/3B2yyCzSyGzSKCsUuCtnrvfU6HCC4X4HaGN41JJO4N7OLeIN8b6CUSnL3d+68simlGIQdixpgYwIsAlgCoB7CLMfYx5/xIqOceDiHmh/v0fQhwe85+ci+ebcaGNWk4tluN6YvPBukzHWZwzkOebx2OgcPSn726D8B01J14CcC9fp8zNT8l7O2YNjolqEBcMsfsLe6xMTaHpz1u4Jv3UpBf5H1DjmXX39eG0weVWP3rbDzwh9qIBo1gHNqixhvPZEMq9+C+/6s/b86CkMRib237ybPMMHaJsXuDFkd2qLH1Ex02fuD9W8nIc6Cg1IrCEgsKS6xIzXIF9YHC7QYMHRJ0t3m/ulqlaK2TobFKjuYzMrgc3iAlEnNk5DtQWGpFdkE3Mkc5kDnKgdQsJ8QC7oYpk3PI5C4kpw//HJyjvxfrcnp7sa7enitj3p49Y+jvrQPo/74v2PYHXwmCev3vWzh2+A0PUjh6xDMBnOKcVwEAY+xNANcCiHogZkzYQCyXiJGXosSZjrNDvPlFNmiSXTiy3TcQm+1utBntyNAqot7OvkStR5eVweWwA1gNoBH7vn4e+75+HhKZHM+sPdh/fLpGhlFpqrC3IzdZiSydAs1+doDyZ+DwtCjGcv4rt2nQ1iDDHY/H/vIghdqDb/93M156JA9/eSwX33+2Hqok4dKCPR7gi3+lYf3qNOQX2bDyfxrjJiEqKcWNRTd3YdHNXXA5GOpOylFVqUR1pdI7t9ybpa7RuaBM8kAm90Cm4JDKvUOwMoUHUgWHVOaBqVuM7jYputskMHRI4PH4/iJpdC7kFNox7+pu5BTakVPoQGa+IyLbasYCxgCJFJBIud914YkiHIE4F0DdOd/XA5gVhvMGLUurgEom7LR3oV7jE4hFIu8nZ/9VtsyCBOK+HvETr27AR3/5P+z7eiGAbyCVK1A6bwmuueenPseXh3FueKCpo5Lx2aHmIR8/ZYF3eLrmiAKFJbHVK/763RSkZjlQepGw64aHavREG1auasTfVuXgb0/m4nv/r16QNzurSYTV/+edD555eQ9u/GGroAVQQiGRcRQU27zD/bd2wePxLg2sqlSi4bQCdguD0y6CwyaC3SqCsUsEh53BYRPBaWdQ69xI1rswrtyCZL0LyXoXUjKc3tsZLijVtIYqEUUtajHG7gFwDwCMGjUqItcQsjfcp1CvxlfHfO+bPMvst8pWTYcZswrToto+p9uDNqN3SY02LQMMhQByIBJvhcthh0Kl8dlHWSEVY1K2NmLtmZCRhM2K9iEX+CidZ4Jc6cGOz3QxFYirDytQc0SJ63/QKuhwYLCKKiz49mPNePWX2XjlqRzc9YvoVgFrrJbh1Z/noKNZihvvb8HcGJoPDgeRyLuGO6fQAaBH6OaQGBWOwb0GAPnnfJ/Xe58PzvnLnPMKznmFXq8f+HBYRGO3pQvRKqRIH1A1q2i6d27z6E7f9jX32H2WO0VDc48NnnNSlVvqvR+K7nj8esxdtgLGLt9SnSW5WkiHWMZyOEQiFlQ2tlzJMXWRAfs3JsFqip2x6a/eSYUqyY2Zl8Xfm23ZRSYsf7gZJ/ep8a9fZcMdhRFhzoGta3X43Q9HwWYR4fvP1GPeNYkVhAkZqnC8k+0CMJ4xVsAYkwFYDuDjMJw3KEkKCTKSoj/M68/YAT1zuZKjsMSKY7t97/dwHvUqWwM3esgpuAtqnQul8/Jw4w9XYeWqF/ofEzEW0WHpPiW5uqD2Lp6ztAdOuwh7voyNzTNa66U4vE2NeVd3x+081owlRtxwfwsqt2mw5jdZEa0iZTWL8K//zca7f8jE2FIrfvLnMyiM8eQ2QiIp5EDMOXcBuB/AFwCOAnibc3441PMGKxaGpfsMLHcJABNnmNFcI0dXq+9sQE17tAPx4IpahaX+C9yPzVBDq4h8tQKFVIzJQQx/50+wI3ecDds+TY6JkpffvJcCsYTjomu7hW5KSC66pgdLV7Zj75davP9CRkRe29rjcjz3/VE4uEmDq+5sw92/aoiZjG1ChBKWsT3O+aec8wmc87Gc81+F45zBiqVAnKmVQyP3DbiTZngD7sBe8ZkOC3iUognnHI3nLF3qapWgs0UacKlNOAt4XMjUUclBDUvOWdqDpirv7jlCMnaJsWudFhWXGhIioFyyvBOLbunE1rXJ+OSVENadDMA58M17yfjjj0fB42b4wW/rcMnyrpjLfCdECAnxZyAVM4xKDf/ymuFijA36YJA52oHkdCeO7fJtp8XhRoshOvWIO80Onznp0we9Fcj8rdXM0MqRlxK91zRZJfM7khDItEVGyOQewTcw2PLvZLicIiy8KXYLeASDMWDZd9sxd1k3vnwrFRvWhL5+3NQjwt//Jwcf/SUDk2aa8dCfzsRcERFChJQQgTg/RTXkfXGjZeB+yIx5h6dP7FMNSoapjtLw9KEG30Si04dUUGrcyB4z+INANHvDfaYFsSuTQu3BlIVG7PsqCTaLMBk+DhvDlo+TUTzH5LOpR7xjDLjh/lZMW2zAp//Q44WH87DzCy3s1uBeZ6tJhN0bkvDb+0bj+F4Vrv9+K1auavSpMEcISZBAHEvD0n1GpaogFfu+cU2cYYHdIkb1Ed9a2NFI2LI63KgcEIirDipRUGwdVOpOJROjSIDym3kpKmQGsa56ztIeOGwi7PsqcsurzmfnOi3MBjEW3RxbmzuEg0gErPhJM5bd1QZjpwRv/jYLq24dizXPZuLUAWXAZK7uNgm2fKzDn3+WiydvHos3nsmGTMHxwO/qMP+6bsqKJsSPuN/0oampCcuXL8dbb72FrKwsoZvTTyIWYVSaGqfPKeE4YaoFIjHHsV1qjDtnOLjFYIPF4YpoMZJ9tV0++w8bOsRoa5Bh9tLBy21Kc3WCjTBMHZWMzyuHVuBj1EQbsgvs2PapDnOuiu6yob5ylqMnWRN2mFUsARbf4q0YVXNEgV3rtNj3TRJ2rdchNcuBGUsMmLHEAIdNhENbNajcqkHdCe8HKX2eAwtu7ELpXBNGTbTRXDAh5xH3fx6/+MUvsHnzZvz85z8XuimDFPb21A0drXjh4W/BYW9BQbEVR3f59uA5B2raI7O/KgDYXW7sr+/2ue/0ob75Yd/rikUMZQIMS/eZkJk0KNEtEMaA2Ut7UH9SgfqT8gs/IYwObdGgo0mGhTd1JXwvjzGgoNiGW37ciqffrMLtP21CWrYT615Pwy/vKMQz94zBZ/9MBxNxXHVnG376t2o89koNrr6rHWMmUxAm5ELitkesVCphs53tibz00kt46aWXoFAoYLXGxprEQr0ajAHrVv8J1ZW7se71FzFxxu/xyd/16OkQQ5d2NnGqpsOMyTmRGWI9WN8Du9N3LPH0QRXkSg9yx/nOD4/P0Aw5EEaCWMQwLkOD/XXdQzp++mID/v3XdGz/TIebxrdGtnG9OAe+eicFaTkOlM6Nj3KW4SJTnN0SsLNFgv3fJEGh8qB4jsnn95kQMnRx+1m1qqoKt912G1Qqb2avSqXC7bffjurqaoFbdlaaLgk/XlKErWvXgHOOrWvX4JO/XwIAOLZr8DImlzv8SSwutwf7agdn9FYdUmJMsXVQOcYpQSRMRUowc/6qJA+mXGzEni+Tgk4mGq5ju1WoPa7Ewhu6orZJfSxKzXRh8S1dmLush4IwISGI20CcnZ0NrVYLm80GhUIBm80GrVYbU/PEVVVVuPyaGyGVe+fNpHIFpi4ag6QU+6BAbHO6caTJEPY2HG40wGz3fZM09YjQfEaOsaW+w9JJCgmydb6JZELIS1EGVWlr9tIe2C1i7P8m8glmbhfw8V/0SM9xYNaV8VfOkhASe+I2EANAS0sL7r33Xmzfvh333nsvmpuHvotPNGRnZyNbnwqXww6JTA6Xww6lWoNJM204vlcF94BOxJ4zXWEt7uHxcOw+46837B1FGLh+OD9G1mJLxKKg1oUXFNuQOcoelTXFWz9JRkutHNd8rw2SyBcdI4SMAHE7RwwA77//fv/tF198UcCWBNbT2Y6F196OaZffjO2fvgVDZxumLfLuxnTmqO9Wft0WJ061mjA+TEuHjjUbYbAOXt96+pASEpkH+RN8s31jqShKQboap1qHNv/KGDD7yh589JcMNFbLkFPgiEibLEYRvngtDeOnmlE8O7qlSQkhiSuuA3E8eP/99/H18Vbsq+3GjT9cBQCwmiwQibzLmAZu5bf7TFdYAjHnHLvP+F/fWnVQiTGTbIN6dLEUiPsS3YY6QFCxxIC1r6Rj+6c63PCDtoi0ad3rabCaRbj2e20JnylNCImeuB6ajheF6b6lG5UaD8ZMHryMCfBuU1jXGfpSptNtJnSYBvcMrSYRGqvkg5YtpWtkUAuYLT2QSiZBVhDFPdRaD8ouMmHPf7Rw2MMfJVvrpNj8cTJmX9nTu7csIYSEBwXiKMhLUUIu9X2pJ86woOGUAobOwWm3gXqywdhV47/2cVWlEpyzmJ0fPlewFdPmLO2B1STGwU1Dr1k9VB+/rIdM7sEVd3SE/dyEkJGNAnEUiEQMY9J8g8rE3t2Yju8eHGxq2i1oMw5/I4jaDguae/xXezp9UAmx1INRE30fj8lArA8uEI8ts0Kf68C2T8ObtHV8twpHdmiw5PbOhNhhiRASWygQR8nATSByx9qRlOrC0V3+A+CeEHrFO2sCP7fqkBKjimyQyc9OvooYQ16K8MuWBspIUiBJMfThcsaAWVf2oLpShZZaWVja4HYDH/1Fj7QcB+bH+X7DhJDYRIE4Skanqn0SfBgDJlaYcXyvetAyJgA43myCwRb8jj5NPdaAc8w2C0P9ScWg/YezdHLIJbFZmWLgB5gLmbHEALHUgw/+pB+0y9VwbP9Uh+YzclxzdxsksujsG00IGVkoEEeJUiYetLPQxAozrEYxao8NTkrycI69ftYAX8jO6sC94ZojSng8bFAgjsVh6T4F6cHN9yaluHHTD1txYq8a7/4xc8hZ1/5YTSJ8/mo6xpVbUDKXlisRQiKDAnEUjU7zDXgTplnARBzH/MwTA96qWDbn0Ock20328+5tXHVICZGIY0yxbyCOpWVLA+UHWWULAGZdYcClKzqw4zMdvnxr+Bvbr1udCotRhGvvbaXlSoSQiKFAHEUDE7bUWg9GT7QNKnfZx+Hy4MAQNz8w2JxYd7jlvD3A0wdVyBtvg1x59iCZRBQTZS0DkYhFw+qxX/mdDkxbZMAnr+ix76vg12W31kux6cMUzLqiB7ljabkSISRyKBBHUZZWMWgZ06QZZtSdUMDY5X+Odn9d9wU3gzjdZsLq7bVoMZzNhO7betHQ6S1u0dMhxpljCoyb4tsbzk1WQiyK7e5eYZDLmADvHPzyh1tQWGLBG7/JRFXl0NckA8C//6qHVO7Bld+h5UqEkMiiQBxFIhEbNAzcv4xpj/9en8UReDMIt4fjmxNt+Hh/46Ah7HO3XgS8m9hz7l1re65Ynh/uU5CuHtbQsETGsfKpRqRmuvDKqly01l+4OLTDxvCfN1NweJsGS26j5UqEkMijQBxl5w5PGzpa8eGfb4Fa5wg4PA343wyix+rEO7vrBiV0PbqsDA9d5rv14kOXzcTX70owZYERUlmDT085lueH+6jlkkGJbkN+rtaDu3/ZACbi+OsTuTB1+x95sFsZvnw7Bb+8owCfvKLHxAozLr6uO4RWE0LI0FAgjrJzE7bWrf4Tag7vgkqzA8f3qOEJ0Pnqtjhx8pwNEE63mfDGjlo0+Sna8cSrGzBt0TKfrRdzCv8AQIvFt3T59JTVcjHSNeFZbxtpwxme7pOe48R3n25ET7sErzyV41MC02YWYcOaVPzy24VY+zc9csfacf9ztbjnfxtouRIhJCpYOLfdG6qKigq+e/fuqF83VsjkCjgd51bOWgHgDYgl8/Hsp3/3+5xMrQK3zsjH5lPtF1zW9O7vV2Hbp29BLJXB5RBBKm+Ey7ETnF8+6FiFQgGr1ernLLGl1WjD6u21IZ3jwCYNXvtlNsrmm3DTj1qw+eNkbHw/BVaTGJNnmbDktk6MnuS/IhkhZGS5b+FYKKThq6/AGNvDOa/w9xj1iAXw1n92+/RaJbKNADy46NrVAZ/TYrDhX9tqhrS22NjdgbnLVuCB37+NwtIX4LQn447Hswf1lK+6/mZUV1eH5WeKtGCrbPlTPt+Eq+9ux4GNSXhq+Vh88Vo6xpZa8eMXzuCuXzRSECaECCJ2ttsZQaZPKoRCpYHLYYdEJofb2Qht6nEc2jwOF1+3E6uf+THuePx5aFP1Ps/rsgyt0tbKVS8AADxuoKdjDEYVWVE2X46T+85e0+WwIys9FVlZWWH/+SKlUK/GgbqeCx94Hgtu7ILVJEJ7kxSLb+lC7tjh1/QmhJBwoB6xAHKSFTD3nO21zl22AimZb6CzRYo3flPpk+0cioObNeholGHxrV1gzLenvPC629HZ3hqGnyZ6gq2y5Q9j3jXG336smYIwISQm0ByxQD7a34CqtrNVsB65qgxu51YAWgATAXgztyQyOZ5ZezDo83MOPPeDUXDYRPjpX2sgGjDVUZanwyWTMof/AwjA5fbgz9+chtNNSVSEkMiiOeIRYPSAKltPvrYBhSWfARgHYDmkcgWmLb4aT7z2n2Gd/8ReFRpOKbDo5s5BQRiIj2VLA0nEIoxKG372NCGExCIKxAIZPSAQatMykDnqMICDAJ6E0+6EQqUZNE88VF++nQJtmgsVlxgHPcZYfBTy8CeUZUzxRCOXYHSaCiIqck1IwqNkLYGkqGVIVknRfU4ClqmnHUXTv8LxPQ9gwrTnYez697DOXXdCjpP71Fh2l/+t+zKSFGEdcommvipbAsyoRBRj3iVqBelqFKarkdFbwKTL7MCumk4cbTLCk2g/NCEEAAViQY1OU6HbcjYLeOWqF+DxAM/eY4eh4x785C+D1/0OxZdvpUKhdmPuVf4zjONxWLpPX5WtZj/FTOKNTCLCqFQVCtLVKEhXQy0f/OeYopbhsuIszCpMw+6aThxpNMDloYBMSCKhoWkBDZwnBgCRCLj0tk40n5Hj0Jbgs4TbGqQ4uFmDeVd3Q6H2v1lEPAdiwNsrjnd5KUrcc3Ehri7PQUmuzm8QPpdOKcUlkzLxnXljMHVUMqRiGrImJFFQIBZQforK785HUxcYoc9zYP3q1KCHYL96JwViCcfF13f7fVwiYshJHl7d5lhRqI/vQKxPkuOaKTmQioP/80tSSLGwKAN3XlSAqaOSw984QkjUUSAWkEwiQk7y4L2ARWLg0uWdaKxS4PD2oQcdQ4cYu9ZrMeMyQ8Bdg3KSlZAMIwDEkowkBbTKC++kFIt0Simun5oLuSS0OXqVTIKFRRkoy9OFqWWEEKHE9ztyAhiT5n+YeNpiA9KyHVj3etqQe8UbP0yBx82w8KbAZTBHBbhevInHXrFaLsYN03IvOAwdjIVFGcj182GOEBI/KBALzN88MQCIJcAlyztRf1KBY7suHDy7WiXY+m8dyueboM8NXAoz3ueH+4wNQ5WtaJJJRLhuSi6SVeHd7UosYriqLDvkOtyEEOFQIBaYPkkOTYAeUsWlBqRkOLFudeBescvB8J+3UvB/d42By8VwyfKOgNdSSMXISJKHo9mCy0tRQi6Nj19fiYjhmvKc/iVJ4aaWS3BVWTYkfvINCCGxLz7eyRJcoOFiidTbKz5zVImT+wYfc3SXCs/eOxqf/F2PCVMt+OlfzyB3rCPgdcZnaMASpECESMRQEAdVthgDrijJingBlWydEosmZkT0GoSQyKBAHAPGnCegzLzMAF26E1+8fjaDuqNJgldW5eCvj+cBHLj7V/W48+lGpGUHHpJmDJg+OiXcTRdUoT72h6cXFWVgfGZSVK5VkqtDeT4lbxESb2hiKQb0lTL0VzlJIuNYfEsXPvhTBo7tUuHMMSW+fCsFIjFw1XfbsOD6br/VswYq1GuQog7v/KTQxqR7l3+5Y7TAxezCNJTnJ0f1mgsmZKDd6EBDtzWq1yWEDB/1iGOAQipGpjbw3O2sK3uQlOrCX5/Iw7rX01B6kQk/e6UGl9zaNaQgDAAVCdYbBgC5RIy8lNjMGC5IV2PO2LSoX5eStwiJPxSIY0Sg7GkAkMk5rrm7DYWlFnz/N3X49mPNSE53DfncOckKv+uVE0EsDk+LGMP88emCXV8tl2BZWQ4lbxESJygQx4gx6edP5pl+iRH3/7Ye48qCH3KcPjp1uM2KebG4nrgkV4s0jbDZ6Vk6BSVvERInKBDHiCytAvphLC0ydLTihYe/BUNnm9/HU1RSjI3BYBUuWoUUGecZ1o82mUSE2YXRH5L2pyRXF5MfVAghvigQxwjG2HnnFAMF3HWr/4Tqyt1Y9/qLfp83fXRqwixZCqQwhop7VIxOCWvlrFAtmKCnIWpCYlxIgZgxdjNj7DBjzMMYqwhXo0aqsXpNwA0ZBgbcR5eV4aHLirB17RpwzrF17Ro8dFkRHl1W1v8clUyMSdnRWTojpFjp8ScpJJgWY0lxySpZzLWJEOIr1B5xJYAbAGwMQ1sIgLljfZN8AgVczjmmLVoGqdwbuKVyBaYtvhpPvPaf/udOyU+O+w0ehiJDq4iJLOE5Y9OGtaNSpM0YkxoTrw8hxL+Q3jU450c558fD1RgC5KeqfOpBP/HqBr8B98l/fQmFSgOXww6JTA6Xww6FSgNtqh6Ad64y2mtYhTRW4OzpDK0ck7O1grYhEJlEhIsEzOImhJxf1D6+M8buYYztZoztbmvzn1hEvOaNO/umqU3LCBhwjd0dmLtsBR74/duYu2wFjF3t/c+bnKOFQhraVnvxROikpIvH62N6Ln5ilpZ2aSIkRl1wvIoxtgFAlp+HHuecfzTUC3HOXwbwMgBUVFTEZimkGJGlU2BshganW00A0B9wZy+9Fds/fas/YWvlqhf6n3PjD1f13xYxhmn5I2teMC9FBZlEBIfLE/VrF+rVEa8lHQ4Li/R4Y2ftkLfVJIRExwUDMef80mg0hPiaOzYNVW0mcB444AYyPlMDnUoayebFHLGIoSBdjePNxqheV8QYLhoXH8O+GVoFSnJ0ONTQI3RTCCHniL3MEgIASNfIMTFreBnPiba5w1AJMTwdC8U7gjFvXPqImrIgJB6EunzpesZYPYA5AD5hjH0RnmYRAJhTmA5xkGtA81NVyIzQvrexbkyaOujXKxQyiUiQetKhUMrEmF2YuJXWCIlHoWZNf8A5z+OcyznnmZzzy8PVMALoVFIU5wSXiTtSe8OAd/OMaCYkVYxOgUoWf8uCyvOSka5JrJ24CIlnNDQd42YVpkEqHlovL10jQ0F6bBS3EEq0hqdjsXjHUIlEDAuLqA41IbGCAnGM08glKMtLPu8xIsYwPlODy0v8JbePLNHajWneuPSYLN4xVPmpKozPjJ3SoISMZPE3rjYCzRiTikMNPYOW5silIpTk6FCenwydcmRlSQeiU0qhT5KjzWiP2DWydYphJ9LFkvnj9ahpN8PpTuz1TIwBDAweWrdFYhQF4jiglIkxbVQKtld1APDuqDRlVAomZ2shk8RvryxSCvXqiAVixoAFRbFdvGOodEopZhemYdPJ9gsfHKNEjGF0mgpZOgUUUjEUUhEUEnH/bbnE+6/bw9FmsqO5x4YWgw3NPTZ0W520pprEBArEcWLa6GS0m+woztGiIF2dEIEgUiZmabGruisiPaCJWUnI1iVOharpo1NQ1WZGQ3fw+1wLKV0jw6RsLSZla4e025VEzJCtU/r839mcbrQYbGjqsaGpx4raDiv1mokgKBDHCblEjKvLc4RuRlxIVctQlqfD/rrusJ5XJhH5lB9NBIwxXFacidU7agWpShYMhVSMoiwNJmfrkKULfYmeQirG6DQ1Rqd5E/wMNicq63tQ2dgDs90d8vkJGSoKxCQhzRmbhuMtRlgd4XtDnT46BUmKxJuLT1bJMH98Ov5ztFXopviVm6zElFHJKExXR3Q3Ma1Cirnj0jGrMA2n20w4WN+Duk5LxK5HSB8KxCQhKaRizClMw5fHwhNckhSShF6jXZaXjNNtJtS0x07gkUlEuGhcOsrydFGdihGLGCZkJmFCZhI6zQ4cqO/G0SYD7M7YHjEg8YsyfUjCKs3VIT0pPOUn54/Xx/VypaG4dFJmzJS/LEhX49tzRqM8P1nQfIhUtQyLijJw9/xCXDIpA6lqKoRCwi+x31nIiCYSMSycoA/5PLnJShQlwHKlC0lSSLFoYuivVyiUMjEuL87CdVNzoY2haQCpWISyvGTcMWc0rpua67NnOCGhoqFpktD6ClecbDEN6/l9y5VGiolZWpxuNeNES3R3sQKACZlJWDRRH9NlQxnz7vJVkO5dIrevtgvHm41weeIn25ox79SNsm+5l1R8zvdiDBz4GZhI7vZwONweOFweON0e2F3e2333OVwe2JweykAPQuz+xhMSJqEUrpicrR1xm2gsnpiBhm5L1DKHNXIJFk3UY1xGfI066JPkuKw4CxeNT8eBuh4cbuyB0eYSulkAvPPrySopUlQyJCulSFbJkKKWIlkpg0IqivhwP+ccdpcHdqcHNpcbNqcbVqcbNqcHNqe7P3jbXe7ef32/j0aRGRFjEIu8I2dixiAWsd77GES9t6OFAjFJeDqlFNNGp2BHVWdQz0vE5UpDoZSJcemkTHy0vzHi1xqfqYmpuenhUMkkmDM2DXPGpqHdZEdNuxnV7WY0dtsi3iuUS0VIV8uRppEhXSNHqlqGFLUMmiGsrY4kxlh/T1uH4KcYOOdwezjcff96ODwewOXxwM29t/viZH+47P/ee0PcG2CZCP2BlrFzb8dOLQYKxGREmDEmFUcaDUH1WGYWpA6pWEQiKtRrUJyjxeFGQ0TOL5OIsLBIj+IcXUTOL5R0jRzpGjkqxqTC5nSjttOC6nYzznSYhz3CwBigkomhkUuRopIiPUmONLUM6UnymJpHDyfGGCRiNmIC1Ej5OckIJxWLMH+8Hp8eahrS8TqlFFPzkyPbqBi3oEiPui4rDFZnWM+bk6zAFcXZ0KkSM4j0UUjF/cugOOfosjhhcbj6h2ztLu8Qrfd7N5xuDoVUhCSFFBq5BEkK75dGLono+mkiPArEZMQoykrCgfpuNHSdv5yjPkmOBRP0I/7NTy4R49opOdhwpAVNPbaQzydiDDMLUjGrIBUiUewMC0YDYwypahktfyJ+USAmI8rCIj3e2FE7KBNULRejKEuLydla6MO09jgRpGvkuHVGPg43GrD5VPuwK5Ulq6S4oiQroep0ExIuFIjJiJKRpEBJjg6HGnogETEU6jWYlJ2EMWnqEddLGyrGGEpydRiXocGWU+041NAz5F2LRIxhUnYSFhZl0E5hhARAgZiMOPPGpUOfJEdRVlJcZ+tGm0IqxiWTMlGSq8OXx1rRHGC4WiOXYFSaCmPS1BidpqLXmJALoEBMRhylTIzyEZ6IFYpMrQLLZ+SjssGALafb4XB5kK1TYEy6N/BmJI2sddeEhIoCMSEkaIwxlObpMD5TA8a8iV2EkOGhQEwIGTYadiYkdJQ9QQghhAiIAjEhhBAiIArEhBBCiIAoEBNCCCECokBMCCGECIgCMSGEECIgCsSEEEKIgCgQE0IIIQKiQEwIIYQIiAIxIYQQIiAKxIQQQoiAKBATQgghAqJATAghhAiIAjEhhBAiIArEhBBCiIAoEBNCCCECokBMCCGECIgCMSGEECIgCsSEEEKIgCgQE0IIIQKiQEwIIYQIiAIxIYQQIiAKxIQQQoiAQgrEjLFnGWPHGGMHGWMfMMaSw9QuQgghZEQItUe8HkAJ57wMwAkAj4XeJEIIIWTkCCkQc87Xcc5dvd9uB5AXepMIIYSQkSOcc8R3Avgs0IOMsXsYY7sZY7vb2trCeFlCCCEkfkkudABjbAOALD8PPc45/6j3mMcBuACsDnQezvnLAF4GgIqKCj6s1hJCCCEJ5oKBmHN+6fkeZ4x9B8AyAJdwzinAEkIIIUG4YCA+H8bYFQAeBbCAc24JT5MIIYSQkSPUOeIXACQBWM8Y288Y+3MY2kQIIYSMGCH1iDnn48LVEEIIIWQkospahBBCiIAoEBNCCCECokBMCCGECIgCMSGEECIgCsSEEEKIgCgQE0IIIQKiQEwIIYQIiAIxIYQQIiAKxIQQQoiAKBATQgghAqJATAghhAiIAjEhhBAiIArEhBBCiIAoEBNCCCECokBMCCGECIgCMSGEECIgCsSEEEKIgCgQE0IIIQKiQEwIIYQIiAIxIYQQIiAKxIQQQoiAKBATQgghAqJATAghhAiIAjEhhBAiIArEhBBCiIAoEBNCCCECokBMCCGECIgCMSGEECIgCsSEEEKIgCgQE0IIIQKiQEwIIYQIiAIxIYQQIiAKxIQQQoiAKBATQgghAqJATAghhAiIAjEhhBAiIArEhBBCiIAoEBNCCCECokBMCCGECIgCMSGEECIgCsSEEEKIgCgQE0IIIQKiQEwIIYQIiAIxIYQQIiAKxIQQQoiAQgrEjLFfMMYOMsb2M8bWMcZywtUwQgghZCQItUf8LOe8jHM+BcBaAP8TepMIIYSQkSOkQMw5N5zzrRoAD605hBBCyMgiCfUEjLFfAbgDQA+ARSG3iBBCCBlBGOfn78QyxjYAyPLz0OOc84/OOe4xAArO+aoA57kHwD293xYBOD6sFvuXDqA9jOcbieg1DA96HUNHr2Ho6DUMXbhfw9Gcc72/By4YiIeKMTYKwKec85KwnDC4a+/mnFdE+7qJhF7D8KDXMXT0GoaOXsPQRfM1DDVrevw5314L4FhozSGEEEJGllDniH/NGCsC4AFwBsC9oTeJEEIIGTlCCsSc8xvD1ZAQvSx0AxIAvYbhQa9j6Og1DB29hqGL2msYtjliQgghhASPSlwSQgghAor7QMwYu4Ixdpwxdoox9jOh2xNvGGOvMMZaGWOVQrclXjHG8hljXzHGjjDGDjPGHhC6TfGGMaZgjO1kjB3ofQ2fFrpN8YoxJmaM7WOMrRW6LfGKMVbDGDvUW755d8SvF89D04wxMYATAJYAqAewC8AKzvkRQRsWRxhjFwMwAXhNiKVniYAxlg0gm3O+lzGWBGAPgOvo93DoGGMMgJpzbmKMSQFsBvAA53y7wE2LO4yxhwBUANByzpcJ3Z54xBirAVDBOY/KWux47xHPBHCKc17FOXcAeBPeZVRkiDjnGwF0Ct2OeMY5b+Kc7+29bQRwFECusK2KL9zL1PuttPcrfnsJAmGM5QG4CsDfhG4LGbp4D8S5AOrO+b4e9AZIBMQYGwNgKoAdAjcl7vQOqe4H0ApgPeecXsPg/Q7Ao/AuKSXDxwGsY4zt6a0KGVHxHogJiRmMMQ2A9wA8OGBDFDIEnHN3705ueQBmMsZoqiQIjLFlAFo553uEbksCuIhzPg3AlQB+0DuFFzHxHogbAOSf831e732ERFXvvOZ7AFZzzt8Xuj3xjHPeDeArAFcI3JR4Mw/ANb3zm28CWMwYe13YJsUnznlD77+tAD6Adxo0YuI9EO8CMJ4xVsAYkwFYDuBjgdtERpjeRKO/AzjKOX9O6PbEI8aYnjGW3HtbCW8CJpXMDQLn/DHOeR7nfAy874Vfcs6/JXCz4g5jTN2bdAnGmBrAZQAiuqokrgMx59wF4H4AX8CbIPM25/ywsK2KL4yxNQC2AShijNUzxr4rdJvi0DwA34a3B7K/92up0I2KM9kAvmKMHYT3A/Z6zjktvyFCyASwmTF2AMBOAJ9wzj+P5AXjevkSIYQQEu/iukdMCCGExDsKxIQQQoiAKBATQgghAqJATAghhAiIAjEhhBAiIArEhBBCiIAoEBNCCCECokBMCCGECOj/Awo9HxiErO/ZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# The test data is 50 equally-spaced points from [0,5]\n",
    "x_test = torch.linspace(0, 5, 50)\n",
    "\n",
    "# Put the model into evaluation mode\n",
    "model.eval()\n",
    "likelihood.eval()\n",
    "\n",
    "# The gpytorch.settings.fast_pred_var flag activates LOVE (for fast variances)\n",
    "# See https://arxiv.org/abs/1803.06058\n",
    "with torch.no_grad(), gpytorch.settings.fast_pred_var():\n",
    "    # Obtain the predictive mean and covariance matrix\n",
    "    f_preds = model(x_test)\n",
    "    f_mean = f_preds.mean\n",
    "    f_cov = f_preds.covariance_matrix\n",
    "    \n",
    "    # Make predictions by feeding model through likelihood\n",
    "    observed_pred = likelihood(model(x_test))\n",
    "    \n",
    "    # Initialize plot\n",
    "    f, ax = plt.subplots(1, 1, figsize=(8, 6))\n",
    "    # Get upper and lower confidence bounds\n",
    "    lower, upper = observed_pred.confidence_region()\n",
    "    # Plot training data as black stars\n",
    "    ax.plot(x_train.numpy(), y_train.numpy(), 'k*')\n",
    "    # Plot predictive means as blue line\n",
    "    ax.plot(x_test.numpy(), observed_pred.mean.numpy(), 'b')\n",
    "    # Shade between the lower and upper confidence bounds\n",
    "    ax.fill_between(x_test.numpy(), lower.numpy(), upper.numpy(), alpha=0.5)\n",
    "    ax.set_ylim([-3, 3])\n",
    "    ax.legend(['Observed Data', 'Mean', 'Confidence'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "official-northeast",
   "metadata": {},
   "source": [
    "The black stars in the above plot represent the training (observed) data, while the blue line and the shaded area represent the mean and the confidence bounds respectively. Notice how the uncertainty is reduced close to the observed points. If more data points were added, we would see the mean function adjust itself to pass through these points and the uncertainty would reduce close to the observations."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "gpytorch",
   "language": "python",
   "name": "gpytorch"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}