tutorial_pytorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with PyTorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Ignacio Cases\"\n",
    "__version__ = \"CS224U, Stanford, Spring 2019\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Getting Started](#getting-started)\n",
    "    1. [Importing PyTorch](#importing-pytorch)\n",
    "    1. [Tensors](#tensors)\n",
    "    1. [GPU Computation](#gpu-computation)\n",
    "1. [Neural Networks Foundations](#foundations)\n",
    "    1. [Automatic Differentiation](#automatic-differentiation)\n",
    "    1. [Modules](#modules)\n",
    "    1. [Criteria and Loss Functions](#criteria-loss)\n",
    "    1. [Optimization](#optimization)\n",
    "1. [Training a Simple Model](#model-training)\n",
    "1. [Reproducibility](#reproducibility)\n",
    "1. [References and Further Reading](#references)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started <a id=\"getting-started\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch is a Python package designed to carry out scientific computation. We use PyTorch in a range of different environments: to develop our models in our local machines, up to large scale deployment for training on big clusters, and even to perform _inference_ in embedded, low-power systems. While similar in many aspects to NumPy, PyTorch enables us to perform fast and efficient training of deep learning and reinforcement learning models not only in the CPU but also in a GPU or other ASICs (Application Specific Integrated Circuits) for AI, such as Tensor Processing Units (TPU)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing PyTorch <a id=\"importing-pytorch\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial assumes a working install of PyTorch using the `nlu` kernel, but the content applies to any regular install of PyTorch. If you don't have a working installation of PyTorch, please follow the instructions on the course repo.\n",
    "\n",
    "To get started working with PyTorch we simply begin by importing the torch module,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Side note**: why not `import pytorch`? The name of the package is `torch` for historical reasons: `torch` is the orginal name of the ancestor of the PyTorch library that got started back in 2002 as a C library with Lua scripting. It was only much later that the original `torch` was ported to Python. The PyTorch project decided to prefix the Py to make clear that this library refers to the Python version, as it was confusing back then to know which `torch` one was referring to. All the internal mentions to the library use just `torch`. It's possible that PyTorch gets renamed at some point as the original `torch` is no longer maintained and there is no longer confusion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can verify the version installed and whether or not we have a GPU-enabled PyTorch install by issuing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorch version 1.0.1.post2\n",
      "GPU-enabled installation? True\n"
     ]
    }
   ],
   "source": [
    "print(\"PyTorch version {}\".format(torch.__version__))\n",
    "print(\"GPU-enabled installation? {}\".format(torch.cuda.is_available()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch has a good [documentation](https://pytorch.org/docs/stable/index.html) but it can take some time to familiarize with the structure of the pacakge; we really recommended that you familiarize yourself with it. We will also make use of other imports:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors <a id=\"tensors\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensors collections of numbers represented as an array, and are the basic building blocks in PyTorch.\n",
    "\n",
    "You are probably already familiar with several types of tensors:\n",
    "    \n",
    "- A scalar, a single number, is a zero-th order tensor.\n",
    "    \n",
    "- A column vector $v$ of dimensionality $d_c \\times 1$ is another (very special) type of tensor of order 1.\n",
    "    \n",
    "- A row vector $x$ of dimensionality $1 \\times d_r$ is another (also very special) type of tensor of order 1.\n",
    "    \n",
    "- A matrix $A$ of dimensionality $d_r \\times d_c$ is yet another type of tensor of order 2.\n",
    "    \n",
    "- A cube $T$ of dimensionality $d_r \\times d_c \\times d_d$ is also a tensor of order 3. An image can be thought of a tensor of order 3, where we use the first two dimensions encode pixel luminosity for a given channel that is specified by the third dimension (the color plane).\n",
    "\n",
    "For our purposes, tensors are the fundamental blocks that carry information in our mathematical model, and they are composed using several operations creating a mathematical graph in which information can flow (propagate) forward (functional application) and backwards (using the chain rule). \n",
    "\n",
    "We have seen multidimensional arrays in Numpy: although they are not called the same way, these NumPy objects are also a representation of tensors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Side note**: what is _really_ a tensor? Tensors are important mathematical objects with applications in multiple domains in Mathematics and Physics. The term tensor comes from the usage of these mathematical objects to describe the stretching of a volume of matter under *tension*. They are central objects of study in a subfield of Mathematics known as Differential Geometry, that deals with the geometry of continuous vector spaces. As a very high-level summary (and as first approximation), tensors are defined as multi-linear \"machines\" that a number of slots (their order, aka rank), taking a number of \"column\" vectors and \"row\" vectors *to produce a scalar*. For example, a tensor $\\mathbf{A}$ (represented by a matrix with rows and columns that you could write in a sheet of paper) can be thought of having two slots. So when $\\mathbf{A}$ acts upon a column vector $\\mathbf{v}$ and a row vector $\\mathbf{x}$, it returns a scalar.\n",
    "    \n",
    "$\\mathbf{A}(\\mathbf{x}, \\mathbf{v}) = s$\n",
    "    \n",
    "If $\\mathbf{A}$ only acts on the column vector, for example, the result will be another column tensor $\\mathbf{u}$ of one order less than the order of $\\mathbf{A}$. Thus, when $\\mathbf{v}$ acts is similar to \"removing\" its slot: \n",
    "\n",
    "$\\mathbf{u} = \\mathbf{A}(\\mathbf{v})$\n",
    "\n",
    "The resulting $\\mathbf{u}$ can later interact with another row vector to produce a scalar or be used in any other way. \n",
    "\n",
    "This can be a very powerful way of thinking about tensors as their slots can guide you when writing the code, specially given that PyTorch has a _functional_ approach to modules where this view is very much highlighted. As we will see below, these simple equations above have an completely straighforward representation in the code. At the end, most of what our models will do is to process the input using this type of functional application so that we end up having a tensor output and a scalar value that measures how good our output is with respect to the real output value in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tensor Creation <a id=\"tensor-creation\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get started with tensors in PyTorch. The framework supports eight different types (Lapan 2018):\n",
    "\n",
    "- 3 float types (16-bit, 32-bit, 64-bit): `torch.FloatTensor` is the class name for the commonly used 32-bit tensor.\n",
    "- 5 integer types (signed 8-bit, unsigned 8-bit, 16-bit, 32-bit, 64-bit): common tensors of these types are the 8-bit unsigned tensor `torch.ByteTensor` and the 64-bit `torch.LongTensor`.\n",
    "\n",
    "There are three fundamental ways to create tensors in PyTorch (Lapan 2018):\n",
    "\n",
    "- Call a tensor constructor of a given type, which will create a non-initialized tensor. So we then need to fill this tensor later to be able to use it.\n",
    "- Call a built-in method in the `torch` module that returns a tensor that is already initialized.\n",
    "- By using the PyTorch-NumPy bridge."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calling the constructor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first create a 2 x 3 dimensional tensor of the type `float`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[4.2460e+13, 4.5911e-41, 4.2460e+13],\n",
      "        [4.5911e-41, 0.0000e+00, 0.0000e+00]]) torch.Size([2, 3])\n"
     ]
    }
   ],
   "source": [
    "t = torch.FloatTensor(2, 3)\n",
    "print(t, t.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we specified the dimensions as the arguments to the constructor by passing directly the numbers -- and not a list or a tuple, which would have very different outcomes as we will see below! We can always inspect the size of the tensor using the `size()` method.\n",
    "\n",
    "The constructor method allocates space in memory for this tensor. Note however that the tensor is *non-initialized* for our purposes in Deep Learning. In order to initialize it we need to call any of the tensor initialization methods built-in in the basic tensor types. For example, the tensor we just created has a built-in method `zero_()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zero_()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The underscore after the method name is important: it means that the operation happens _in place_, this is, the returned object is the same object but now with different content.\n",
    "A very handy way to construct a tensor using the constructor happens when we have available the content we want to put in the tensor in the form of a Python iterable: in this case we just pass it as the argument to the constructor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2., 3.],\n",
       "        [4., 5., 6.]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "torch.FloatTensor([[1, 2, 3], [4, 5, 6]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calling a method in the torch module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A very convenient way to create tensors, in addition to using the constructor method, is to use one of the multiple methods provided in the `torch` module. In particular, the `tensor` method allows us to pass a number or iterable as the argument to get the appropriately typed tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A 64-bit integer tensor: tensor([1, 2, 3]), torch.LongTensor\n",
      "A 32-bit float tensor: tensor([1., 2., 3.]), torch.FloatTensor\n"
     ]
    }
   ],
   "source": [
    "tl = torch.tensor([1, 2, 3])\n",
    "t = torch.tensor([1., 2., 3.])\n",
    "print(\"A 64-bit integer tensor: {}, {}\".format(tl, tl.type()))\n",
    "print(\"A 32-bit float tensor: {}, {}\".format(t, t.type()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can create a similar 2x3 tensor to the one above by using the `torch.zeros()` method passing a sequence of dimensions to it: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0., 0., 0.],\n",
      "        [0., 0., 0.]])\n"
     ]
    }
   ],
   "source": [
    "t = torch.zeros(2, 3)\n",
    "print(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can find a large variety of methods to create tensors there. We list some useful ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0., 0., 0.],\n",
      "        [0., 0., 0.]])\n",
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.]])\n",
      "tensor([[5., 5., 5.],\n",
      "        [5., 5., 5.]])\n",
      "tensor([[0.7590, 0.1273, 0.4451],\n",
      "        [0.6683, 0.0196, 0.0866]])\n",
      "tensor([[-1.7895, -0.7535, -0.4807],\n",
      "        [ 0.5964,  1.4192, -0.5725]])\n"
     ]
    }
   ],
   "source": [
    "t_zeros = torch.zeros_like(t)            # zeros_like returns a new tensor\n",
    "t_ones = torch.ones(2, 3)                # creates a tensor with 1s\n",
    "t_fives = torch.empty(2, 3).fill_(5)     # creates a non-initialized tensor and fills it with 5\n",
    "t_random = torch.rand(2, 3)              # creates a uniform random tensor\n",
    "t_normal = torch.randn(2, 3)             # creates a normal random tensor with the specified dimensions\n",
    "\n",
    "print(t_zeros)\n",
    "print(t_ones)\n",
    "print(t_fives)\n",
    "print(t_random)\n",
    "print(t_normal)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now see emerging two important paradigms in PyTorch: an imperative approach to performing operations, using _inplace_ methods, is in marked contrast with an additional paradigm also used in PyTorch, the _functional_ approach, where the returned object is a copy of the original object. Both paradigms have their specific use cases as we will be seeing below. The rule of thumb is that _inplace_ methods are faster and don't require extra memory allocation in general, but they can be tricky to understand (keep this in mind regarding the computational graph that we will see below). _Functional_ methods make the code referentially transparent, which is a highly desired property that makes easier to understand the underlying math, but we rely on the effiency of the implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "t1 = torch.clone(t)      # creates a new copy of the tensor that is still linked to the computational graph (see below)\n",
    "assert id(t) != id(t1), 'Functional methods create a new copy of the tensor'\n",
    "\n",
    "# To create a new _independent_ copy, we do need to detach from the graph\n",
    "t1 = torch.clone(t).detach()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using the PyTorch-NumPy bridge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A quite useful feature of PyTorch is the almost seamless integration with NumPy that allows us to perform operations on NumPy and interact from PyTorch with the large number of NumPy libraries as well. Converting a NumPy multi-dimensional array into a PyTorch tensor is very simple: we only need to call the `tensor` method with NumPy objects as the argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NumPy array: [1. 2. 3.], type: float64\n",
      "Torch tensor: tensor([1., 2., 3.], dtype=torch.float64), type: torch.float64\n"
     ]
    }
   ],
   "source": [
    "# Create a new multi-dimensional array in NumPy with the np datatype (np.float32)\n",
    "a = np.array([1., 2., 3.])\n",
    "\n",
    "# Convert the array to a torch tensor\n",
    "t = torch.tensor(a)\n",
    "\n",
    "print(\"NumPy array: {}, type: {}\".format(a, a.dtype))\n",
    "print(\"Torch tensor: {}, type: {}\".format(t, t.dtype))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert a PyTorch tensor into a NumPy array also seamlessly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1., 2., 3.])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Side note**: why not `torch.from_numpy(a)`? The `from_numpy()` method is depecrated in favor of `tensor()`, which is a more capable method in the torch package. `from_numpy()` is only there for backwards compatibility. It can be a little bit quirky, so I recommend using the newer method in PyTorch >= 0.4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Indexing works as expected with NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.7830,  1.1622])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = torch.randn(2, 3)\n",
    "t[:,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch also supports indexing using long tensors, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.5929,  0.0931, -0.3181,  1.0272, -0.3327,  0.3270],\n",
      "        [ 0.1875, -0.8570, -0.6022, -0.3721, -0.1816, -1.3812],\n",
      "        [ 0.5273, -0.9770,  1.2352,  0.7568, -0.0677, -1.0433],\n",
      "        [-0.3046,  0.4589,  0.1333, -0.3969,  0.0188,  0.1917],\n",
      "        [ 0.9501,  0.9675,  1.9239,  1.1949,  1.2277,  0.0470]])\n",
      "tensor([[ 0.1875, -0.8570, -0.6022, -0.3721, -0.1816, -1.3812],\n",
      "        [-0.3046,  0.4589,  0.1333, -0.3969,  0.0188,  0.1917]])\n",
      "tensor([-0.1816,  0.1917])\n"
     ]
    }
   ],
   "source": [
    "t = torch.randn(5, 6)\n",
    "print(t)\n",
    "i = torch.tensor([1, 3])\n",
    "j = torch.tensor([4, 5])\n",
    "print(t[i])                          # selects rows 1 and 3\n",
    "print(t[i, j])                       # selects (1, 4) and (3, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Type conversion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each tensor has a set of convenient methods to convert types. For example, if we want to convert the tensor above to a 32-bit float tensor, we use the method `.float()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.5929,  0.0931, -0.3181,  1.0272, -0.3327,  0.3270],\n",
      "        [ 0.1875, -0.8570, -0.6022, -0.3721, -0.1816, -1.3812],\n",
      "        [ 0.5273, -0.9770,  1.2352,  0.7568, -0.0677, -1.0433],\n",
      "        [-0.3046,  0.4589,  0.1333, -0.3969,  0.0188,  0.1917],\n",
      "        [ 0.9501,  0.9675,  1.9239,  1.1949,  1.2277,  0.0470]])\n",
      "tensor([[-0.5929,  0.0931, -0.3181,  1.0272, -0.3327,  0.3270],\n",
      "        [ 0.1875, -0.8570, -0.6022, -0.3721, -0.1816, -1.3812],\n",
      "        [ 0.5273, -0.9770,  1.2352,  0.7568, -0.0677, -1.0433],\n",
      "        [-0.3046,  0.4589,  0.1333, -0.3969,  0.0188,  0.1917],\n",
      "        [ 0.9501,  0.9675,  1.9239,  1.1949,  1.2277,  0.0470]],\n",
      "       dtype=torch.float64)\n",
      "tensor([[  0,   0,   0,   1,   0,   0],\n",
      "        [  0,   0,   0,   0,   0, 255],\n",
      "        [  0,   0,   1,   0,   0, 255],\n",
      "        [  0,   0,   0,   0,   0,   0],\n",
      "        [  0,   0,   1,   1,   1,   0]], dtype=torch.uint8)\n"
     ]
    }
   ],
   "source": [
    "t   = t.float()   # converts to 32-bit float\n",
    "print(t)\n",
    "t   = t.double()  # converts to 64-bit float\n",
    "print(t)\n",
    "t   = t.byte()    # converts to unsigned 8-bit integer\n",
    "print(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Operations on Tensors <a id=\"operations-on-tensors\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to create tensors, let's create some of the fundamental tensors and see some common operations on them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(42)\n"
     ]
    }
   ],
   "source": [
    "# Scalars \n",
    "s = torch.tensor(42) # creates a tensor with a scalar (zero-th order tensor, i.e. just a number)\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip**: a very convenient way in newer PyTorch versions is to access to scalars with `.item()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.item()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see higher-order tensors -- remember we can always inspect the dimensionality of a tensor using the `.size()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row vector tensor([[ 0.9739, -1.7025,  0.1483]]) with size torch.Size([1, 3])\n",
      "Column vector tensor([[0.6065],\n",
      "        [1.5179],\n",
      "        [1.0098]]) with size torch.Size([3, 1])\n",
      "Matrix tensor([[-0.1813, -0.5609,  0.4626],\n",
      "        [-1.0871, -0.4376, -0.9243],\n",
      "        [ 0.5843, -0.7621, -0.6608]]) with size torch.Size([3, 3])\n"
     ]
    }
   ],
   "source": [
    "# Row vector\n",
    "x = torch.randn(1,3)\n",
    "print(\"Row vector {} with size {}\".format(x, x.size()))\n",
    "\n",
    "# Column vector\n",
    "v = torch.randn(3,1)\n",
    "print(\"Column vector {} with size {}\".format(v, v.size()))\n",
    "\n",
    "# Matrix\n",
    "A = torch.randn(3, 3)\n",
    "print(\"Matrix {} with size {}\".format(A, A.size()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A common operation is matrix-vector multiplication (and in general tensor-tensor multiplication). For example, the product $\\mathbf{A}\\mathbf{v} + \\mathbf{b}$ is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.4942],\n",
      "        [-2.2569],\n",
      "        [-1.4696]])\n",
      "tensor([[-1.2610],\n",
      "        [-2.3814],\n",
      "        [-1.8043]])\n"
     ]
    }
   ],
   "source": [
    "u = torch.matmul(A, v)\n",
    "print(u)\n",
    "b = torch.randn(3,1)\n",
    "y = u + b                    # we can also do torch.add(u, b)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where we retrieve the expected result (a column vector of dimensions 3x1). We of course can compose operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.143073558807373\n"
     ]
    }
   ],
   "source": [
    "s = torch.matmul(x, torch.matmul(A, v))\n",
    "print(s.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many functions implemented for every tensor, and we encourage you to study the documentation. Some of the most common ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1, 2],\n",
       "        [0, 1, 2]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# common tensor methods (they also have the counterpart in the torch package, e.g. as torch.sum(t))\n",
    "t = torch.randn(2,3)\n",
    "t.sum(dim=0)                 \n",
    "t.t()                   # transpose\n",
    "t.numel()               # number of elements in tensor\n",
    "t.nonzero()             # indices of non-zero elements\n",
    "t.view(-1, 2)           # reorganizes the tensor to these dimensions\n",
    "t.squeeze()             # removes size 1 dimensions\n",
    "t.unsqueeze(0)          # inserts a dimension\n",
    "\n",
    "# operations in the package\n",
    "torch.arange(0, 10)     # tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
    "torch.eye(3, 3)         # creates a 3x3 matrix with 1s in the diagonal (identity in this case)\n",
    "t = torch.arange(0, 3)\n",
    "torch.cat((t, t))       # tensor([0, 1, 2, 0, 1, 2])\n",
    "torch.stack((t, t))     # tensor([[0, 1, 2],\n",
    "                        #         [0, 1, 2]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GPU computation <a id=\"gpu-computation\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deep Learning frameworks take advantage of the powerful computational capabilities of modern graphic processing units (GPUs). GPUs were originally designed to perform frequent operations for graphics very efficiently and fast, such as linear algebra operations, which makes them ideal for our interests. PyTorch makes very easy to use the GPU: the common scenario is to tell the framework that we want to instantiate a tensor with a type that makes it a GPU tensor, or move a given CPU tensor to the GPU. All the tensors that we have seen above are CPU tensors, and PyTorch has the counterparts for GPU tensors in the `torch.cuda` module. Let's see how this works.\n",
    "\n",
    "A common way to explicitly declare the tensor type as a GPU tensor is through the use of the constructor method for tensor creation inside the `torch.cuda` module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.],\n",
       "        [0., 0., 0.]], device='cuda:0')"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t_gpu = torch.cuda.FloatTensor(3, 3)   # creation of a GPU tensor\n",
    "t_gpu.zero_()                          # initialization to zero"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Devices <a id=\"devices\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, a more common approach that gives us flexibility is through the use of devices. A device in PyTorch refers to either the CPU (indicated by the string \"cpu\") or one of the possible GPU cards in the machine (indicated by the string \"cuda:$n$\", where $n$ is the index of the card). Let's create a random gaussian matrix using a method from the `torch` package, and set explicitly the computational device to be the GPU by specifying the `device` to be `cuda:0`, our first GPU card in our machine (this code will fail if you don't have a GPU, but we will work around that below): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.5961,  0.3101, -0.6615],\n",
       "        [ 1.1299,  1.2280, -1.0802],\n",
       "        [ 0.5733,  0.8676, -0.6882]], device='cuda:0')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t_gpu = torch.randn(3, 3, device=\"cuda:0\")\n",
    "t_gpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can notice, the tensor now has the explicit device set to be a CUDA device, not a CPU device. Let's now create a tensor in the CPU and move it to the GPU:\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.0262,  0.2573, -1.2256],\n",
       "        [-0.5341, -1.2264,  0.3200],\n",
       "        [ 0.9489, -0.6399, -0.8682]])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = torch.randn(3, 3)   # we could also state explicitly the device to be the CPU with torch.randn(3,3,device=\"cpu\")\n",
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that in this case the device is the CPU, but PyTorch does not explictly say that given that this is the default behavior. To copy the tensor to the GPU we use the `.to()` method that every tensor implements, passing the device as an argument. This method creates a copy in the specified device or, if the tensor already resides in that device, it returns the original tensor (Lapan 2018): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-1.0262,  0.2573, -1.2256],\n",
      "        [-0.5341, -1.2264,  0.3200],\n",
      "        [ 0.9489, -0.6399, -0.8682]], device='cuda:0')\n",
      "cuda:0\n"
     ]
    }
   ],
   "source": [
    "t_gpu = t.to(\"cuda:0\")  # copies the tensor from CPU to GPU\n",
    "# note that if we do now t_to_gpu.to(\"cuda:0\") it will return the same tensor without doing anything else as this tensor already resides on the GPU\n",
    "print(t_gpu)\n",
    "print(t_gpu.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip**: When we program PyTorch models, we will have to specify the device in several places (not so many, but definitely more than once). A good practice that is consistent accross the implementation and makes the code more portable is to declare early in the code a device variable by querying the framework if there is a GPU available that we can use. We can do this by writing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda:0\n"
     ]
    }
   ],
   "source": [
    "device = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device(\"cpu\")\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use `device` as an argument of the `.to()` method in the rest of our code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.0262,  0.2573, -1.2256],\n",
       "        [-0.5341, -1.2264,  0.3200],\n",
       "        [ 0.9489, -0.6399, -0.8682]], device='cuda:0')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.to(device)   # moves t to the device (this code will **not** fail if the local machine has not access to a GPU)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Side note**: having a good GPU backend support is a critical aspect of a Deep Learning framework -- even so that some models depend crucially on performing computations on a GPU. Most frameworks, including PyTorch, only provide good support for GPUs manufactured by Nvidia. This is mostly due to the heavy investment this company made on CUDA (Compute Unified Device Architecture), the underlying parallel computing platform that enable this type of scientific computing (and the reason for the device label), with specific implementations targeted to Deep Neural Networks as cuDNN. Other GPU manufacturers, most notably AMD, are making efforts to towards enabling ML computing in their cards, but their support is still partial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Network Foundations <a id=\"foundations\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Computing gradients is a crucial feature in deep learning, given that the training procedure of neural networks rely on optimization techinques that update the parameters of the model by using the gradient information of a scalar magnitude -- the loss function. How is it possible to compute the derivatives? There are different methods, namely\n",
    "\n",
    "- **Symbolic Differentiation**: given a symbolic expression, the software provides the derivative by performing symbolic transformations (e.g. Wolfram Alpha). The benefits are clear, but it is not always possible to compute an analytical expression.\n",
    "- **Numerical Differentiation**: computes the derivatives using expressions that are suitable to be evaluated numerically, using the finite differences methods to several orders of approximation. A big drawback is that these methods are slow.\n",
    "- **Automatic Differentiation**: it is an approach in which a library adds to the set of functional primitives (a number of functions for which an implementation is available) an implementation of the derivative for each of these functions. Thus, if the library contains the function $sin(x)$, it also implements the derivative of this function, $\\frac{d}{dx}sin(x) = cos(x)$. Then, given a composition of functions, the library can compute the derivative with respect a variable by succesive application of the chain rule, a method that is known in deep learning as backpropagation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automatic Differentiation <a id=\"automatic-differentiation\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modern deep learning libraries are capable of performing automatic differentiation, although the underlying implementation model result in very different user experiences. The two main approaches to computing the graph are _static_ and _dynamic_ processing (Lapan 2018):\n",
    "\n",
    "- **Static graphs**: the deep learning framework converts the computational graph into a static representation that cannot be modified. This allows the library developers to do very aggressive optimizations on this static graph ahead of computation time, pruning some areas and transforming others so that the final product is highly optimized and fast. The drawback is that some models can be really hard to implement with this approach. One of the libraries we use in the course, TensorFlow, uses static graphs. Having static graphs is part of the reason why TensorFlow has excellent support for sequence processing, which makes it very popular in NLP.\n",
    "\n",
    "- **Dynamic graphs**: the framework does not create a graph ahead of computation, but records the operations that are performed, which can be quite different for different inputs. When it is time to compute the gradients, it unrolls the graph and perform the computations. A major benefit of this approach is that implementing complex models can be easier in this paradigm (e.g. conditional computation is really easy). This flexibility comes at the expense of the major drawback of this approach: speed. Dynamic graphs cannot leverage the same level of ahead-of-time optimization, which makes them slower. PyTorch uses dynamic graphs as the underlying paradigm for gradient computation.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Simple Graph](fig/simple_computation_graph.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Simple graph to compute $y = wx + b$ (from Rao and MacMahan 2019) \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch computes the graph using the Autograd system. Autograd records a graph when performing the forward pass (function application), keeping track of all the tensors defined as inputs. These are the leaves of the graph. The output tensors are the roots of the graph. By navigating this graph from root to leaves, the gradients are automatically computed using the chain rule. In summary,\n",
    "\n",
    "- Forward pass (the successive function application) goes from leaves to root. We use the apply method in PyTorch.\n",
    "- Once the forward pass is completed, Autograd has recorded the graph and the backward pass (chain rule) can be done. We use the method `.backwards()` on the root of the graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modules <a id=\"modules\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The base implementation for all neural network models in PyTorch is the class `Module` in the package `torch.nn`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All our models subclass this base `nn.Module` class, which provides an interface to important methods used for constructing and working with our models, and contain sensible initializations for our models. Modules can contain (and usually do) other modules. \n",
    "\n",
    "Let's see a simple, custom implementation of a multi-layer feed forward network. In the example below, our simple mathematical model is\n",
    "\n",
    "$\\mathbf{y} = \\mathbf{U}(f(\\mathbf{W}(\\mathbf{x})))$\n",
    "\n",
    "where $f$ is a non-linear function (a `ReLU`), is directly translated into a similar expression in PyTorch. To do that, we simply subclass `nn.Module`, register the two affine transformations and the non-linearity, and implement their composition within the `forward` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyCustomModule(nn.Module):\n",
    "    def __init__(self, n_inputs, n_hidden, n_output_classes):\n",
    "        super(MyCustomModule, self).__init__()         # call super to initialize the class above in the hierarchy\n",
    "        self.W = nn.Linear(n_inputs, n_hidden)         # first affine transformation\n",
    "        self.f = nn.ReLU()                             # non-linearity (here it is also a layer!)\n",
    "        self.U = nn.Linear(n_hidden, n_output_classes) # final affine transformation\n",
    "        \n",
    "    def forward(self, x):\n",
    "       y = self.U(self.f(self.W(x)))\n",
    "       return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we can use our new module as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.5364, -0.1848]], grad_fn=<AddmmBackward>)\n"
     ]
    }
   ],
   "source": [
    "# set the network's architectural parameters\n",
    "n_inputs = 3\n",
    "n_hidden= 4\n",
    "n_output_classes = 2\n",
    "\n",
    "# instantiate the model\n",
    "model = MyCustomModule(n_inputs, n_hidden, n_output_classes)\n",
    "\n",
    "# create a simple input tensor \n",
    "x = torch.FloatTensor([[0.3, 0.8, -0.4]]) # size is [1,3]: a mini-batch of one example, this example having dimension 3\n",
    "\n",
    "# compute the model output by **applying** the input to the module\n",
    "y = model(x)\n",
    "\n",
    "# inspect the output\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see, the output is a tensor with its gradient function attached -- Autograd tracks it for us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Tip**: modules overrides the `__call__()` method, where the framework does some work. Thus, instead of calling directly the `forward()` method, apply the input to the model instead."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequential <a id=\"sequential\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A powerful class in the `nn` package is `Sequential`, that allows to express the code above more succintly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyCustomModule(nn.Module):\n",
    "    def __init__(self, n_inputs, n_hidden, n_output_classes):\n",
    "        super(MyCustomModule, self).__init__()\n",
    "        self.network = nn.Sequential(\n",
    "            nn.Linear(n_inputs, n_hidden),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(n_hidden, n_output_classes))\n",
    "        \n",
    "    def forward(self, x):\n",
    "       y = self.network(x)\n",
    "       return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can imagine, this can be handy when we have a large number of layers for which the actual names are not that meaningful. It also improves readability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyCustomModule(nn.Module):\n",
    "    def __init__(self, n_inputs, n_hidden, n_output_classes):\n",
    "        super(MyCustomModule, self).__init__()\n",
    "        self.p_keep = 0.7\n",
    "        self.network = nn.Sequential(\n",
    "            nn.Linear(n_inputs, n_hidden),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(n_hidden, 2*n_hidden),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(2*n_hidden, n_output_classes),   \n",
    "            nn.Dropout(1 - self.p_keep),       # dropout argument is probability of dropping\n",
    "            nn.Softmax(dim=1)                  # applies softmax in the data dimension\n",
    "        )\n",
    "        \n",
    "    def forward(self, x):\n",
    "       y = self.network(x)\n",
    "       return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Side note**: Another important package in `torch.nn` is `Functional`, typically imported as `F`. Functional contains many useful functions from non-linear activations to convolutional, dropout, and even distance functions. Many of these functions have counterpart implementations as layers in the `nn` package so that they can be easily used in pipelines like the one above implemented using `nn.Sequential`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0., 5.]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "y = F.relu(torch.FloatTensor([[-5, -1, 0, 5]]))\n",
    "\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Criteria and Loss Functions <a id=\"criteria-loss\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch has implementations for the most common criteria in the `torch.nn` package. You may notice that, as with many of the other functions, there are two implementations of loss functions: the reference functions in `torch.nn.functional` and practical class in `torch.nn`, which are the ones we typically use. Probably the two most common ones are (Lapan 2018):\n",
    "\n",
    "- `nn.MSELoss` (mean squared error): squared $L_2$ norm used for regression.\n",
    "- `nn.CrossEntropyLoss`: criterion used for classification as the result of combining `nn.LogSoftmax()` and `nn.NLLLoss()` (negative log likelihood), operating on the input scores directly. When possible, we recommend using this class instead of using a softmax layer plus a log conversion and `nn.NLLLoss`, given that the `LossSoftmax` implementation guards against common numerical errors, resulting in less instabilities.\n",
    "\n",
    "Once our model produces a prediction, we pass it to the criteria to obtain a measure of the loss:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(0.5327, grad_fn=<NllLossBackward>)\n"
     ]
    }
   ],
   "source": [
    "y_gold = torch.tensor([1])        # the true label (in this case, 2) from our dataset wrapped \n",
    "                                  # as a tensor of minibatch size of 1\n",
    "criterion = nn.CrossEntropyLoss() # our simple classification criterion for this simple example\n",
    "\n",
    "y = model(x)                      # forward pass of our model (remember, using apply instead of forward) \n",
    "\n",
    "loss = criterion(y, y_gold)       # apply the criterion to get the loss corresponding to the pair (x, y)\n",
    "                                  # with respect to the real y (y_gold)\n",
    "\n",
    "print(loss)                       # the loss contains a gradient function that we can use to compute\n",
    "                                  # the gradient dL/dw (gradient with respect to the parameters for a given fixed input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization <a id=\"optimization\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have computed the loss for a training example or minibatch of examples, we update the parameters of the model guided by the information contained in the gradient. The role of updating the parameters belongs to the optimizer, and PyTorch as a number of implementations available right away -- and if you don't find your preferred optimizer as part of the library, chances are that you will find an existing implementation. Also, coding your own optimizer is indeed quite easy in PyTorch.\n",
    "\n",
    "**Side Note** The following is a summary of the most common optimizers. It is intended to serve as a reference (I use this table myself quite a lot). In practice, most people pick an optimizer that has been proven to behave well on a given domain, but optimizers are also a very active area of research on Numerical Analysis, so it is a good idea to pay some attention to this subfield. We recommend to use second-order dynamics with adaptive time step (following Fedkiew 2019):\n",
    "\n",
    "- First-order dynamics\n",
    "    - Search direction only: `optim.SGD`\n",
    "    - Adaptive: `optim.RMSprop`, `optim.Adagrad`, `optim.Adadelta`\n",
    "    \n",
    "- Second-order dynamics\n",
    "    - Search direction only: Momentum `optim.SGD(momentum=0.9)`, Nesterov, `optim.SGD(nesterov=True)`\n",
    "    - Adaptive: `optim.Adam`, `optim.Adamax` (Adam with $L_\\infty$)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a Simple Model <a id=\"model-training\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to illustrate the different concepts and techniques above, let's put them together in a very simple example: our objective will be to fit a very simple non-linear function, a sine wave:\n",
    "\n",
    "$y = a \\sin(x + \\phi)$\n",
    "\n",
    "where $a, \\phi$ are the given amplitude and phase of the sine function. Our objective is to learn to adjust this function using a feed forward network, this is:\n",
    "\n",
    "$ \\hat{y} = f(x)$\n",
    "\n",
    "such that the error between $y$ and $\\hat{y}$ is minimal according to our criterion. A natural criterion is to minimize the squared distance between the actual value of the sine wave and the value predicted by our function approximator, measured using the $L_2$ norm.\n",
    "\n",
    "**Side Note**: Although this example is easy, simple variations of this setting can pose a big challenge, and are used currently to illustrate difficult problems in learning, especially in a very active subfield known as meta-learning (e.g. Finn et. al 2017, Schulman and Nichol 2018)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's import all the modules that we are going to need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.utils.data as data\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Early on the code, we define the device that we want to use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\") if torch.cuda.is_available() else torch.device(\"cpu\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's fix $a=1$, $\\phi=1$ and generate traning data in the interval $x \\in [0,2\\pi)$ using NumPy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x7ffae0379e48>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3X+UlNWZJ/Dvl6YbkcEFYsOaBgKDfTQaDD1bg3A4J4c1dkRJpCZR0cCMycnAP5OZMeQ4aYQT1EAkQ4LMmZOdXTE/yNCrGKMlWRiZjpHNLguMTRpp0RB+hCiNA51BIgGkoXn2j347qW6r33ur6+db7/dzTp3uqvdW1wWaeure+9zn0swgIiLSa0ipOyAiIuVFgUFERPpQYBARkT4UGEREpA8FBhER6UOBQURE+lBgEBGRPhQYRESkDwUGERHpY2ipOzAYV199tU2aNKnU3RARiZQ9e/b8xsxqXe0iGRgmTZqE1tbWUndDRCRSSP7ap52mkkREpA8FBhER6UOBQURE+lBgEBGRPhQYRESkDwUGERHpQ4FBRET6yMs+BpLfBfBJACfN7CMZrhPAPwC4A8A5AJ8zs58H1+4HsDxoutLMNuSjT+Knce12HDx5NqefMWvKGDQvmpmnHolIqeVrxPB9AHNCrt8OoD64LQbwTwBAcgyAFQBuBjAdwAqSo/PUJxlAqq0DN371RUxq2pJzUACAHYdPYXLTFqTaOvLQOxEptbwEBjP7GYBTIU3mAfiB9dgFYBTJawDcBqDFzE6Z2TsAWhAeYCRHC9bvxAOb9uJsV3def64BeGDTXkxq2oLlqfa8/mwRKa5ilcSoA/BW2v1jwWMDPf4+JBejZ7SBiRMnFqaXFSof00XZ2LjrTWzc9aammEQiqliLz8zwmIU8/v4HzZ4ws4SZJWprnTWgJHDt0vxMFw3GjsOnMKlpC25e1VKS1xeRwSlWYDgGYELa/fEAjoc8Ljm6aUXPGsKljGG2uE6c6cK1S7eUuhsi4qlYgWEzgL9gjxkAfmtmbwPYBuATJEcHi86fCB6THNy04kW8eyG/awi5umTApKYtuPGrL2qRWqTM5Std9SkAswFcTfIYejKNqgHAzP47gK3oSVU9hJ501c8H106R/BqAV4If9aiZhS1iS4gF63dix+Hc/voWzpiIlcmpfR5LtXVgzbYDOH76PGqGDsGFS5cH/fPPdnXjgU170frrU+97HREpDzQrg7mGLCUSCdN5DH3dvKoFJ850Deq5BPD4/GlINmRc9x/Q8lQ7Nu56c1CvCQDrBvGaIjJ4JPeYWcLZToEh2lJtHXjwh3txMcsP8UMIrL0nf2/Mgw1MylwSKR7fwKCSGBGWauvAA5uyDwqzpozBkcfm5vXT+u5ljZg1ZUzWz9tx+JSylkTKjAJDhD2waW9W7WuqiHXzpxXsE3rzopk4unou6seOyOp5J850KTiIlBEFhghKtXXg2oe2ercfXj0E6+ZPwy9X3VGUOf2WJbNxdPXcrEYQJ850oXHt9sJ1SkS8KTBETO/00aXLfmtD40bW4I2v3V6SRd7mRTOxcIb/LvWDJ89iwfqdBeyRiPhQYIiYbKaP6seOwO5ljQXsjdvK5FSsmz8Nw4b6/artOHxKtZZESkyBIUKuX+Y/fTRryhi0LJlduM5kIdlQhwMrb8dVw6q82m/c9aZGDiIlpMAQEY1rt+O9br/po4UzJpZlCui+R+Zg3Mgar7bKVhIpHQWGMpdq68jq3IR186eV9Y7i3csavbOWlK0kUhoKDGWsd6HZ18IZEyOxk7hlyWzvRWkFB5HiU2AoY8ue91+EzVTjqJytTE7F0dVzMbza/St44kyXFqRFikiBoYz5nrJW7tNHYR779E1e7TbuelNVWUWKRIGhDKXaOlD/kN/5BVEvRJdsqPNec3hg014FB5EiUGAoM9nUP4p6UOjVsmS29y/il7IsAyIi2VNgKDO+b3xHV+e3CF6prZ0/zaudASqdIVJgeQkMJOeQPEDyEMmmDNcfJ7k3uP2S5Om0a91p1zbnoz9R1bh2e+YDr/vJtkhdFCQb6rDOMzgcPHlWU0oiBZTzCW4kqwB8G0Ajes5wfoXkZjN7vbeNmX0prf1fA2hI+xHnzczvHaGCLU+1e+1VGDeypmx2NOdb7wjIJ0X3gU17K2rEJFJO8jFimA7gkJkdMbMuAE8DmBfS/j4AT+XhdSuKz0los6aMKXnto0JLNtR5V2W9acWLBe6NSDzlIzDUAXgr7f6x4LH3IfkhAJMB/DTt4StItpLcRTKZh/5Ejm9doHIsc1EIzYtmegWHdy90q6aSSAHkIzAww2MDTZXfC+BZM0tP0J8YHDX3WQDrSE7J+CLk4iCAtHZ2dubW4zKz4/ApZ5tsyldXguZFM73WHHYcPqX1BpE8y0dgOAZgQtr98QCOD9D2XvSbRjKz48HXIwC2o+/6Q3q7J8wsYWaJ2traXPtcNnwybKK2qzlffKeVlj63rwi9EYmPfASGVwDUk5xMsgY9b/7vyy4ieR2A0QB2pj02muSw4PurAcwC8Hr/51aq65dtdS44R3lXcz40L5qJK6oyDUr/4PzFyyqZIZJHOQcGM7sE4IsAtgF4A8AzZraf5KMk70xreh+Ap80sfZrpwwBaSb4K4GUAq9OzmSrZTStedJbRHgIo8wbAL1bd4Wyjkhki+ZNzuioAmNlWAFv7PfbVfvcfzvC8/wcgdh+HF6zfiXcvuOsg+W76ioOFMyY6M7e+/IxSWEXyQTufS8BnsblSyl3ky8rkVOd6Q7f5Z3iJyMAUGMqUgsL7NS+a6czO0pnRIrlTYChDcUtNzcbK5NSM+dHptN4gkhsFhiJasH4nJjWFl9OeNWVMrLOQfCzwCJxrth0oQk9EKpMCQ5EsWL/TubYwvHpIbHY358IncHacPl+EnohUJgWGInEFheoh9D7NTPwqzOqsaJHBUWAoAp/57jV3f1QLzlnwqTB74kyX1hpEBkGBocBSbR348jOvhrapGzVcQWEQjq6ei6uGVYW2eUjlMkSypsBQQL3HdHZb+A7nB2+7rkg9qjz7HpkTev3cxcsaNYhkSYGhgJZ4HDizcMZEjRZy5Pol9jn4R0T+QIGhgC6HXCPiWzU133xKh1y/bKuzjYj0UGAoENd+hcdjXjU1n5INdc5Nb+91m1eJcxFRYCgIn0+nmj7KL59NbwdPntV6g4gHBYYC8CmnLfnlU2QPAP7u2fAMMRHRe1Te+VT3VDntwmheNBPjRtaEtunqNo0aRBzyEhhIziF5gOQhkk0Zrn+OZCfJvcHtL9Ou3U/yYHC7Px/9KZVUW4dXSW1NIxXO7mWNzjbKUhIJl3NgIFkF4NsAbgdwA4D7SN6QoekmM5sW3J4MnjsGwAoANwOYDmAFydG59qlUlj3vLvesyqmFt85jRKZzG0QGlo8Rw3QAh8zsiJl1AXgawDzP594GoMXMTpnZOwBaAITvWCpjZ7vCT2UbN7JGmUhF4DMi8xnZicRVPgJDHYC30u4fCx7r7zMk95F8luSELJ9b9lyfQMeNrPGa5pD88BmZqcieSGb5CAyZUsj7p+X8GMAkM7sJwE8AbMjiuT0NycUkW0m2dnZ2DrqzheCztqCgUFwrk1OdFVhPnOkqUm9EoiUfgeEYgAlp98cDOJ7ewMz+w8wuBHfXA/gvvs9N+xlPmFnCzBK1tbV56HZ+pNo6sOSZ8MXMK6uV/FUKPhVYtelN5P3y8Y71CoB6kpNJ1gC4F8Dm9AYkr0m7eyeAN4LvtwH4BMnRwaLzJ4LHIuORH+/H5fBtC/i6zlkoGVdMPnjyrM6IFukn58BgZpcAfBE9b+hvAHjGzPaTfJTknUGzvyG5n+SrAP4GwOeC554C8DX0BJdXADwaPBYZ75y76Gyj9NTSWXO3O0Np4643i9ATkeigOUpCl6NEImGtra2l7gYa127HwZNnQ9uoUF7p+Ryrqn8niQOSe8ws4Wqnye9BWp5qdwaF+rEj9GZTBnzO0daoQeQPFBgGyeeNxGfxU4rD54xoLUSL9FBgGASfWjs+Bd2keFqWzHaW5naNAEXiQoFhEB78obvWjs/0hRTX4yqVIeJFgSFLqbYOXAw7mg1+tXqk+JINdc4pJZXKEFFgyJpPZU6lp5YvbXoTcVNgyILPNIOqp5Y/17+R1hok7hQYsuCaZqgilJ4aAT7/RtoNLXGmwODJJxPpW/dobSEqXOtA2tcgcabA4OlLjrWFWVPGaG0hQpINdc70VWUoSVwpMHhoXLs9cy3wNEpPjR5X+uqOw6d0PrTEkgKDQ6qtw7kYqc1s0eQzwnvouX1F6IlIeVFgcHBNIQEaLUSZK0Pp3MXLGjVI7CgwhEi1dTinkDRaiDafDKU12w4UoSci5UOBIcSy590pixotRJ9r1HD89Pki9USkPCgwDCDV1oGzXd2hbVT6ojKsTE4NHfkZlKEk8ZKXwEByDskDJA+RbMpwfQnJ10nuI/kSyQ+lXesmuTe4be7/3FJxFcobN7JG6akVpHnRzNCRw47DpxQcJDZyDgwkqwB8G8DtAG4AcB/JG/o1awOQMLObADwL4O/Trp03s2nB7U6UgeWpdmehvN3LGovTGSmalcmpGDW8esDrKrAncZGPEcN0AIfM7IiZdQF4GsC89AZm9rKZnQvu7gIwPg+vWzDNjl2vo68c+M1Dou2358PP8FapDImDfASGOgBvpd0/Fjw2kC8A+Je0+1eQbCW5i2RyoCeRXBy0a+3s7MytxyGWp9qdmUgrPnVjwV5fSuuDo4aHXlepDImDfASGTJUFMr63klwIIAFgTdrDE4PDqT8LYB3JKZmea2ZPmFnCzBK1tbW59nlArv/4C2dM1NpCBXvwtuucbTRqkEqXj8BwDMCEtPvjARzv34jkrQCWAbjTzC70Pm5mx4OvRwBsB9CQhz4Nis9GJlVPrWzJhjpn+upTu98KvS4SdfkIDK8AqCc5mWQNgHsB9MkuItkA4H+gJyicTHt8NMlhwfdXA5gF4PU89GlQvvKj8PIHdY5pBqkMruDfba7JRpFoyzkwmNklAF8EsA3AGwCeMbP9JB8l2ZtltAbAHwH4Yb+01A8DaCX5KoCXAaw2s5IFhguXwlORfKYZpDK4Rg1KXZVKRovgp59EImGtra15/ZmNa7c7i+UdXT03r68p5c31OzFryhjtfJdIIbknWNMNpZ3PAVdQ0JGd8dOyZLZz05tIJVJggHtaYNzIGi06x5Tr312VV6USKTDA/clPu5xlIEt1XoNUoNgHBtcnvqFDXAdASqWrqRr4d+D8xcva1yAVJ/aB4eHN+0Ovf/PujxapJ1Ku/v6u8N8B7YaWShPrwLA81Y7TIbVxhg6hdjmL16Y3pa9KJYltYEi1dTg/6Wm0IL1ci9DKUJJKEtvA8IDjLGfVRJJsaa1BKkUsA4NqIslguEqiaK1BKkUsA4PPWc4i/anyqsRFLAOD6yznsPN/Jb6SDXWodvyP0ahBKkHsAoPPJzrVv5GBrLl7mrONRg0SdbEKDD6ZSCqUJ2GSDXWhG94AjRok+mIVGB78oTsTScTFteFNJOpiExiWp9pxMeS4hRE1VcpEEi/Jhjpc6VhsmNy0pUi9Ecm/vAQGknNIHiB5iGRThuvDSG4Kru8mOSnt2tLg8QMkb8tHfzJxDe9X/ZmCgvj7+qdvCr1uAK5ftrU4nZGKtzzVjilLt2JS0xZMWbq14OtYOQcGklUAvg3gdgA3ALiP5A39mn0BwDtmdi2AxwF8I3juDeg5CvRGAHMA/Lfg5+WVz74FbWaTbCQb6lA/dkRom/e6o3cIlpSfBet3YuOuN39/pGy3GTbuerOgwSEfI4bpAA6Z2REz6wLwNIB5/drMA7Ah+P5ZAB8nyeDxp83sgpn9CsCh4Ofl1ZptB0Kvj76yOt8vKTHQsmS2s40ylCQXqbaOAcutPLX7rYK9bj4CQx2A9B4eCx7L2CY4I/q3AD7g+dycHT99PvT6ik/dmO+XlJhwJSxs3PWmDvORQVsSUrqnu4DHMucjMGTK3evf44Ha+Dy35weQi0m2kmzt7OzMqoMfDCllUD1E00gyeD4JC4/8OLy0u8hAQvJlUMXCnRWTj8BwDMCEtPvjARwfqA3JoQD+E4BTns8FAJjZE2aWMLNEbW1tVh188LbrUD1A7rnPhiWRMK6pyHfODVzaXWQgrmnI+26eEHo9F/kIDK8AqCc5mWQNehaTN/drsxnA/cH3dwH4qZlZ8Pi9QdbSZAD1AP4tD33qI9lQhzV3fbTPf+BRw6uxbv40jRYkZz5TkY1rtxe+I1JRmh2ZlIVMrx+a6w8ws0skvwhgG4AqAN81s/0kHwXQamabAXwHwD+TPISekcK9wXP3k3wGwOsALgH4KzMLL2Q0SMmGOgUBKYhkQx1af30qNCX64MmzReyRRN3yVHvmOfUioRVwAaNQEomEtba2lrobIn1MbtoS+p954YyJ2kQpXiY5NkgO9neJ5B4zS7jaxWbns0ihLfDIUBJxcR0TWz92RME/YCgwiOSJz39WrTWIi+uYWJ/9M7lSYBDJo3Eja0Kva61Bwrg+OLhqdOWLAoNIHu1e1uhsow1vkkmqrcP5wcFVoytfFBhE8mxETXi5L1f5d4kn1+9F/dgRRcusVGAQyTNXpd6w8u8ST65jAYDirC30UmAQybNkQx3WzQ/fUa9FaEnn2sw2bGhx36oVGEQKwDXkP3jyrDMtUeLDtZvsG58pztpCLwUGkQJxnQ294/ApLUSLc/Q4oqaq6FUbFBhECsTnbOiHnttXhJ5IufLJRCrF6ZIKDCIFkmyow6wpY0LbnNNKdKz5fDAoRY03BQaRAmpeNBNDh4RPKemUt/hyfTBwHQRVKAoMIgX2zbvDp5RUQymeXB8IiMKW1g6jwCBSYD5TAUpfjZdUW4fzA8HjjpTnQlJgECmCupDjZQHVUIqbZc+HjxZKfeRwToGB5BiSLSQPBl9HZ2gzjeROkvtJ7iM5P+3a90n+iuTe4KZzNqUiPXjbdc42Sl2Nh+WpdpztCj+PrNRHDuc6YmgC8JKZ1QN4Kbjf3zkAf2FmNwKYA2AdyVFp1x80s2nBTUVkpCIlG+qcC4kPbNKvfxz4rCmV+rTJXAPDPAAbgu83AEj2b2BmvzSzg8H3xwGcBFCb4+uKRI7OaxCfDDRXinMx5BoYxpnZ2wAQfB0b1pjkdAA1AA6nPbwqmGJ6nOSwHPsjUtbqx44IvX7w5Fmlr1Ywn5pIzYtmFqk3A3MGBpI/Iflahtu8bF6I5DUA/hnA582sN3l3KYDrAfwpgDEAvhLy/MUkW0m2dnZ2ZvPSImXDp0Km0lcrV7nVRBqIMzCY2a1m9pEMtxcAnAje8Hvf+E9m+hkkrwKwBcByM9uV9rPfth4XAHwPwPSQfjxhZgkzS9TWaiZKost1yhughehKdNOKF51tSr220CvXqaTNAO4Pvr8fwAv9G5CsAfA8gB+Y2Q/7XesNKkTP+sRrOfZHpOz5nPL28Ob9ReiJFMvyVDvevRCeiVSqXc6Z5BoYVgNoJHkQQGNwHyQTJJ8M2twD4GMAPpchLbWZZDuAdgBXA1iZY39EIsH1JnD6/EWNGiqIa3qwfuyIku1yzoRmrlmv8pNIJKy1tbXU3RDJyeSmLaFzzlUEDj82t2j9kcJYnmp3Boajq4vz70xyj5klXO2081mkRFwlD7pNBfYqgc9oodwoMIiUSLKhDuF1V5WhFHU+gb2YZzn7UmAQKaEFHguOOgI0ulyBfURNVZF6kh0FBpESWpmcCscJoNhx+FRxOiN55ZOeWorT2XwoMIiU2LfucRdMU6mMaEm1dTjTU4dXDymbfQv9KTCIlFiyoc55ypvKckfLl59xF0R87NPlscs5EwUGkTLgOuUN0KghSroduwBmTRlTtqMFQIFBpCwkG+qcpTI0aogGV7LAFVUsi0J5YRQYRMqET6kM7Wsof65kgV+suqNIPRk8BQaRMlLt+B+pfQ3lzRW4XRlo5UKBQaSM+BzpqLWG8pRq63AGbp8MtHKgwCBSRpINdV6H+Uj5cR3NWj92RFkvOKdTYBApMz4lErQburz4jOLKsfTFQBQYRMpQFcMno7Ubury4RnGjhlcXqSf5ocAgUobuu3mCs41GDeXBZ7Tw8J03Fr4jeZRTYCA5hmQLyYPB19EDtOtOO6Rnc9rjk0nuDp6/KTjtTST2VianOtcadhw+pcN8SizV1uG15hOVtYVeuY4YmgC8ZGb1AF4K7mdy3symBbc70x7/BoDHg+e/A+ALOfZHpGK0LJntLMutI0BL66Hn9jnblNORnb5yDQzzAGwIvt+AnnObvQTnPN8C4NnBPF8kDlyH+egI0NI6d/Fy6PVyO7LTV66BYZyZvQ0AwdexA7S7gmQryV0ke9/8PwDgtJldCu4fAxCt8ZZIgSUb6jBrypjQNsue127oUnCV1SailYmUzhkYSP6E5GsZbvOyeJ2JwTmjnwWwjuQUIOMoecDSUyQXB8GltbOzM4uXFom25kUzQ4PD2a5u3LyqpYg9kgXrdzrLartGe+XMGRjM7FYz+0iG2wsATpC8BgCCrycH+BnHg69HAGwH0ADgNwBGkRwaNBsP4HhIP54ws4SZJWpra7P4I4pEX/OimRh95cApjyfOdClLqYh80oWjtuCcLteppM0A7g++vx/AC/0bkBxNcljw/dUAZgF43cwMwMsA7gp7voj0WPGp8JRH7W0oDp8A7Jr+K3e5BobVABpJHgTQGNwHyQTJJ4M2HwbQSvJV9ASC1Wb2enDtKwCWkDyEnjWH7+TYH5GKlWyoc2YpaSG6sJan2p0B+KphVWVfVtuFPR/coyWRSFhra2upuyFSdMtT7aGF2oYAOLJ6bvE6FDNTlm5Ft+M982gZ//2T3BOs94bSzmeRCFmZnBo6argMVV8tJFdQiOKehUwUGEQixpXtcvDkWS1EF4Brmq56CCK5ZyETBQaRiPHJdtFCdH4tT7U7y2r7nKURFQoMIhHkk/WiY0DzY8H6nc4DeEZfWR3p9NT+FBhEIsgn60XHgOYu1dbhHH0R7lTiqFFgEIkon4VOLUTn5svPhE8fEcCCGRMrarQAKDCIRJZPae6DJ89qb8MgLU+1o9uRzf/4/GkVs+CcToFBJMJalsxGlWPXm2vRVDLzmYqrtJFCLwUGkYj71j3ubBhXJVDpyyfdd9jQyn37rNw/mUhMJBvqMHRI+LDh3Qvd2tvgyWfBGQC+8ZmbitCb0lBgEKkA37z7o842OgrUj8/U27r50yp2GglQYBCpCD4H+gDAIz/WUaBhfALnwgrMQupPgUGkQjQvmomrhlWFtnnn3MUi9SaaXKOFIaicshdhFBhEKsi+R+Y421y/bGsRehI91y7d4myzNsKnsmVDgUGkwrimlN7rNkxqcr8Jxsn1y7bikmPPAlG56an9KTCIVJjmRTOdG98A7Yru1bh2O95z7WQD8KsyPmch33IKDCTHkGwheTD4OjpDm/9Kcm/a7T2SyeDa90n+Ku1aPMZpIgXWsmQ2Rg0f+IxooGdXtPj9PayLyRRSr1xHDE0AXjKzegAvBff7MLOXzWyamU0DcAuAcwD+Na3Jg73XzUxbNEXy5OE73YXdJjdtiXUK682rWpxtrhpWFZsppF65BoZ5ADYE328AkHS0vwvAv5jZuRxfV0Qckg11uMJRL8PQk4kTx+CwYP1OnDjTFdpmKP0W9CtNroFhnJm9DQDB17GO9vcCeKrfY6tI7iP5OMlhAz2R5GKSrSRbOzs7c+u1SEz8YtUdXu3iVk9pwfqdXrubDz0Wn3WFdM7AQPInJF/LcJuXzQuRvAbAVADb0h5eCuB6AH8KYAyArwz0fDN7wswSZpaora3N5qVFYs33cHqfaZVK0Lh2u1dQ8NkwWKmcgcHMbjWzj2S4vQDgRPCG3/vGfzLkR90D4Hkz+/0OGzN723pcAPA9ANNz++OISCbDq92TAyfOdFX8qW/LU+3ei+4+hyFVqlynkjYDuD/4/n4AL4S0vQ/9ppHSggrRsz7xWo79EZEMHvu0X8G3Sj/1zffPF7cspP5yDQyrATSSPAigMbgPkgmST/Y2IjkJwAQA/7vf85tJtgNoB3A1gJU59kdEMkg21Hm/2VVqppLPVFn1kMovkOdjaC5PNrP/APDxDI+3AvjLtPtHAbzvb9rMbsnl9UXEX7KhDq2/PuX81GwAvhQsRlfSG6QrAwkADn49novN/Wnns0iMrExO9VpUNVROJdYF63d6lQDxOUM7LhQYRGKmedFMr+DwzrmLkc9U8s1Aqh4Sj6qpvhQYRGLIt57SiTNdkT0WNJsMpDV3x3uxuT8FBpGYalkyG+NG1jjbvXuhO3Klupen2r0zkOJw8E62FBhEYmz3skavaaUoler2DQp1o4Zj3fxpmkLKQIFBJOaaF830TmUt92mlBet3eo8UdjTdopHCABQYRATJhjrvaaVy3ePgu9AMxLvchQ+auQ+oKDeJRMJaW1tL3Q2RinPTihfx7oVuZ7sqAt+6pzw2gjWu3Z712RK+9aMqDck9ZpZwtdOIQUR+z7fEdLf1VGStf6i0u6SvXbpFQaEAFBhEpI9s6gRdvNwTIIpdfG95qh2TmrY4z2lOd9WwKgUFTzmVxBCRytM7PZTNGQ0bd72Jlv3/jt3LGgvVLQBAqq0DSzbtxeUsnxfXA3cGSyMGEXmf3qJ71Y4T4NKdONOFSU1bsGD9zoL0aXmqHQ8MMijE9cCdwVJgEJGMkg11WHPXR7N+3o7DpzApjxVaU20dmNS0ZVAlwceNrFFQGARlJYlIqFRbR05Hfy6cMTHrTWSptg489Nw+nLuY7fggt9etdL5ZSVpjEJFQg1lzSLdx15vv+7Q/ang1Hr7zRiQb6pBq68CabQfQcfp8zn0FekYJhV7rqHQ5jRhI3g3gYQAfBjA9OIchU7s5AP4BQBWAJ82s90CfyQCeRs95zz8H8Odm5iyarhGDSGksWL/TexNZKSgohCvWPobXAHwawM9COlIF4NsAbgdwA4D7SN4QXP4GgMfNrB7AOwC+kGN/RKSAsimfUWyzpoxRUMiTnAKDmb1hZgcczaYDOGRmR4LRwNMA5gXnPN8C4Nmg3Qb0nPssImUs2VCHo6vnlk1ZiYUzJuK2fpueAAAE7ElEQVTo6rloXjSz1F2pGMVYY6gD8Fba/WMAbgbwAQCnzexS2uOl318vIl5634gHU5IiH2ZNGaNgUCDOwEDyJwD+c4ZLy8zsBY/XyJQIbSGPD9SPxQAWA8DEiTqCT6RctCyZDaA46w/1Y0f8/vWkcJyBwcxuzfE1jgGYkHZ/PIDjAH4DYBTJocGooffxgfrxBIAngJ7F5xz7JCJ51vvpPZtDcnxpUbm4irHB7RUA9SQnk6wBcC+AzdaTDvUygLuCdvcD8BmBiEgZW5mciqOr52Ld/GmoGzUcADDEfwN1H8OGDsG6+dMUFIos13TVPwPwjwBqAZwGsNfMbiP5QfSkpd4RtLsDwDr0pKt+18xWBY//Mf6QrtoGYKGZXXC9rtJVRaIrfd9CFYluM9SNGo4Hb7uuLMp4VzLfdFXtfBYRiQmdxyAiIoOiwCAiIn0oMIiISB8KDCIi0ocCg4iI9KHAICIifUQyXZVkJ4BfD/LpV6Nn13WURf3PoP6XXtT/DOr/4HzIzGpdjSIZGHJBstUnj7ecRf3PoP6XXtT/DOp/YWkqSURE+lBgEBGRPuIYGJ4odQfyIOp/BvW/9KL+Z1D/Cyh2awwiIhIujiMGEREJEavAQHIOyQMkD5FsKnV/skXyuyRPknyt1H0ZDJITSL5M8g2S+0n+ban7lA2SV5D8N5KvBv1/pNR9GgySVSTbSP6vUvdlMEgeJdlOci/JyJVZJjmK5LMkfxH8Xyi780ljM5VEsgrALwE0oudUuVcA3Gdmr5e0Y1kg+TEAvwPwAzP7SKn7ky2S1wC4xsx+TnIkgD0AklH5NyBJACPM7HckqwH8XwB/a2a7Sty1rJBcAiAB4Coz+2Sp+5MtkkcBJMwskvsYSG4A8H/M7Mng8LIrzex0qfuVLk4jhukADpnZETPrQs8BQfNK3KesmNnPABT2UN0CMrO3zeznwfdnALwBIDIns1iP3wV3q4NbpD5ZkRwPYC6AJ0vdlzgieRWAjwH4DgCYWVe5BQUgXoGhDsBbafePIUJvSpWG5CQADQB2l7Yn2QmmYfYCOAmgxcwi1X/0nKT4dwAul7ojOTAA/0pyD8nFpe5Mlv4YQCeA7wXTeU+SHFHqTvUXp8CQ6dTZSH3aqxQk/wjAjwA8YGbvlro/2TCzbjObBmA8gOkkIzOlR/KTAE6a2Z5S9yVHs8zsTwDcDuCvginWqBgK4E8A/JOZNQA4C6Ds1jvjFBiOAZiQdn88gOMl6ktsBXPzPwLQbGbPlbo/gxUM/7cDmFPirmRjFoA7gzn6pwHcQnJjabuUPTM7Hnw9CeB59EwTR8UxAMfSRprPoidQlJU4BYZXANSTnBws+NwLYHOJ+xQrweLtdwC8YWZrS92fbJGsJTkq+H44gFsB/KK0vfJnZkvNbLyZTULP7/9PzWxhibuVFZIjgsQFBFMwnwAQmSw9M/t3AG+RvC546OMAyi75YmipO1AsZnaJ5BcBbANQBeC7Zra/xN3KCsmnAMwGcDXJYwBWmNl3SturrMwC8OcA2oN5egB4yMy2lrBP2bgGwIYgw20IgGfMLJIpnxE2DsDzPZ8xMBTA/zSzF0vbpaz9NYDm4APqEQCfL3F/3ic26aoiIuInTlNJIiLiQYFBRET6UGAQEZE+FBhERKQPBQYREelDgUFERPpQYBARkT4UGEREpI//D+h9AeyNoUJ7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "M = 1200\n",
    "\n",
    "# sample from the x axis M points\n",
    "x = np.random.rand(M) * 2*math.pi\n",
    "\n",
    "# add noise\n",
    "eta = np.random.rand(M) * 0.01\n",
    "\n",
    "# compute the function\n",
    "y = np.sin(x) + eta\n",
    "\n",
    "# plot\n",
    "plt.scatter(x,y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the NumPy-PyTorch bridge\n",
    "x_train = torch.tensor(x[0:1000]).float().view(-1, 1).to(device)\n",
    "y_train = torch.tensor(y[0:1000]).float().view(-1, 1).to(device)\n",
    "\n",
    "x_test = torch.tensor(x[1000:]).float().view(-1, 1).to(device)\n",
    "y_test = torch.tensor(y[1000:]).float().view(-1, 1).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SineDataset(data.Dataset):\n",
    "    def __init__(self, x, y):\n",
    "        super(SineDataset, self).__init__()\n",
    "        assert x.shape[0] == y.shape[0]\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.y.shape[0]\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        return self.x[index], self.y[index]\n",
    "\n",
    "sine_dataset = SineDataset(x_train, y_train)\n",
    "sine_dataset_test = SineDataset(x_test, y_test)\n",
    "\n",
    "sine_loader = torch.utils.data.DataLoader(sine_dataset, batch_size=32, shuffle=True)\n",
    "sine_loader_test = torch.utils.data.DataLoader(sine_dataset_test, batch_size=32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SineModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SineModel, self).__init__()\n",
    "        self.network = nn.Sequential(\n",
    "            nn.Linear(1, 5),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(5, 5),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(5, 5),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(5, 1))\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.network(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xl8FdX5+PHPmbtn31eCIRB2InsEQVkEF0QouFWty89q+y1Wu9hWtLVW22pbraVWbdXWilURBVlERWWRPew7hCVAIGTfbpK7zp3z++NCIHBRhCxAzvv14sXNzMmdZ0J45txnzpwjpJQoiqIo7YvW1gEoiqIorU8lf0VRlHZIJX9FUZR2SCV/RVGUdkglf0VRlHZIJX9FUZR2SCV/RVGUdkglf0VRlHZIJX9FUZR2yNzWAZxJQkKCzMzMbOswFEVRLiobNmyokFImflO7Czb5Z2Zmsn79+rYOQ1EU5aIihDh0Nu1U2UdRFKUdUslfURSlHVLJX1EUpR1SyV9RFKUdUslfURSlHVLJX1EUpR1SyV9RFKUdUslfURSlHbpgH/JSzmzX8iUsnzGdusoKIuMTGJ7ZHfOnn6MXF2NOTSXi6quo/2pZ49dJP/0J0ePHt3XYiqJcQFTyv8jsWr6Ez1/7B7rPC0DEngK0fANb7x9jHxSHdFdRt2w2emkRQgqO6Gl8NcuLd8EiIuLsDJnQma65KW18FoqitDWV/C8yy2dMR/d5Wd2tB5uMfozU48mVDjQhABBh8dj734t3o8BfugEjsSdeexw7LTpLAhW4Zldjn+XEkvQ5JGwgtt5C7uHxZNZe1XhxiLwsj4L9z+PxFmO3pZLV+VFSUya08ZkritKcVPK/yDQ44/l4QA4HqjNBWnlI2rEdS/zHaSYL5j63oRetJfHoW6wZUUxe8UgSoj0MChQS4XPT4OuOXp3G1d0/ILLH/3B75lC9+TbeXP0Wi/NX4ZIAdsK1aiZVP8H3BqIuAIpyCRFSyvN/EyH+A9wIlEkpe4fYL4BpwA2AC7hXSrnx695z4MCBUk3s1tSevBL++mU+Cz1FSH80AMtlJEIIPk0x83JXG6V2QbJH8qM9Hvy73DT0fJtOPVYRawpQo2vsONwVeWQQALqALfE7OBixm0R/HHeX38Ro52BAUmvy8rdOhSzI6odEoEmYWOznL70yCe+X1IY/BUVRvo4QYoOUcuA3tWuunv9/gX8A08+w/3og+9ifXODVY38r38Jrn7/N/NwBWBZHAXANZiTwWYqZP/S24zEFPwGUOAR/6OVgVOwKbk1fjvXYmK5Yi8HgzN2sBeSRQZgldK/O5GDkbsqtVUxLexcNGOUcTEzAzq/3dWH4xrmUV27AGRHNvo7deOmLfCyeWiITEhk8ZCIb3Om8lK5RahfEY+Cofx9v9WekhKfwSP9H6B+mqxKSolyAmiX5SymXCSEyv6bJBGC6DH7MWCOEiBFCpEopi5vj+O3Bq0tf5YsO7xAwX4PZ7maMx8yvCNb6X+5qa0z8x3nNgrwO/bmr6WasGvTK2MP2Y73/qCo3N+9LJ9xjAqCMxcxkKVkRlzMw8VpyY/pzb5+5DPIlMrikL++lDqQMgzjhZfdanc6+Gh7eXM2errX8N2cglrBb4NBw0jabObp3A9t6vMsn9YLqgJ1YUxU3Vj3G9wapEpKitLXWqvmnA4dP+vrIsW0q+Z+lL1e+zaQ9KbwzuYb6LjE8vvNLOprfxiQqKLUvCfk9VSIu5PYYs4HdeRnesMMEouIRjkj85UVYnVUASCT76jfzr36ZFGT3p9byJtWeAD7pRJa4AUGltPOxI8Dvwl/ndutC6soT6P3Jnex32DA0H55YF7Mjt1FSacbl6Exd8hTKTfFMC1SwZ9Ms3rheJX9FaUutlfxFiG2n3WwQQjwIPAjQsWPHlo7porHq4xcYvD0Gs6Hxg20vcJ+xmRiLD00Eh3ume0s5Yj99+Ga8rAr5k6/zW/BEHjq2TyCtNryplwE0XgB2dclhW9ccdEuwZlTrMEPvGMyAucSNOWoTlsSF/NFSw5t6Kt+pdlBZryOFQAAOI4wBlQOo9tvJcN7JsuxNbOw9hoA5kQXafcwqqWJySuiLk6IoLa+1kv8RIOOkrzsAR09tJKV8DXgNgjd8Wye0C9+GOYswGxqxHY+QU5CFJdWCZq5r3D+14DUe7fZL3CZ74zZzQGd4cR6BFBMmU6Bxu8+AwwUDT78oaCZ8iemNyX957hh0i7VpG5OG3jUKu2sV9tTZCM0PQLHFzL8SAvSnmI4NwYt2YXgh22O34za7MevPUFd6C/aFRxDCjTl9Eb+u7U7hIcno0aPJyclpxp+Woihno7WS/zzgISHEDII3emtVvf/rbd26lbcX7iXuaCyR5kfxpXrpXvUftkVPZIBpUZO2k8uDX0/N+hlOWzgRXje5BTtIKfez13kFl3XahN3mOjHap6xLyGPKk5K9MyImdGB2E7bEhQjNT5fyAeQW3kiEL5Z6azWb0j8Hgol/Y8JGAlrwoqNbDGLT3mN83RssDIul1qxD7To2+zvjmu8CUBcARWllzZL8hRDvASOABCHEEeC3gAVASvlP4BOCwzz3ERzqeV9zHPdStXXrVt5csIuORxKxBiC5dC1DC+axbeSDHIncT62IJIa6Jt8zuXwRmRVlfBwYjdROdOuryrrQvehGllt2hi6+nUT4fY2vo+prcEbGnt7IE0BYauhSPoCrC27HYgQvGJG+OIYcmoQ7Yj/bO3zSmPg71HWgd3VvwgJh1JlcdG1w0a14ZPCCYathZdd6XqoIkLp8Pb/OzlKlIEVpJc012ue737BfAlOa41jtwaJFi0gqu7wx8XfPfxdbaj/KoooJCINFXMl4vsSK3vg9AQ2s2QeQK0swxySjmzTCpY0cf2fq6hPRom0YZu+ZD2oEsJYXIQEpBB237GT7kCFgOjH3n0mXhO2uQ0bGkFt4Y2PiP85iWNHqO+E2u4Fg4h9QOQCzDP6ahQfCCdcjsWh+BIJIbyyjd4QTL9181iuMn+8+CKAuAIrSCtQTvheg2tpaEoL5k6yCedhS+2Hv9z0axAoAttEDgNGsJJo6amUkR7Il1VHxPJHwBfMS+/LHzg9y1JpM7j4XV21rIKw+k/roPSBOupVy7KVP03HanGSGX8++Th1YkhOGZ3055u016F2jwG4i2mUwcqubPoWSvfHjifCF+FQAaIYNh+7AbXHTu7p3Y+JvJAzcMfn0GvBXzGFV6K447PmTKKoew7ZYM88WFKvkryitQCX/C1B0dDR1tRKH14fdW42t13cQZhsR0k698ADBC8Dxi0A4VgYkvE3i9tvZ1GUHMWlreD7wI7qu70ug5kdo4Q6gA1vDCthi1OD1hmGzNZCZuYmEuEMUfpVKzc5ooIBuFd3oc/AqdtrsfOFqwFcSvArdU2sjWgY/BWRXDsSvebEYttNiDxOCh8pu4++p7xIWCAt5fjoSS3jwxrIlvIrOOdPpWWkmsCuOXT0GNPNPU1GUUFTyv8C8u+Yg07vnEhcj+E4evD/mfuZe1zH4BK1nNP0ObKVz+YlHJkxSo0vyIeK23cG+veVE5azG7jHIOtBAjHsxn6fa6VN5CxZPPD3cPYnq8yremP0ASJeV4sVJ1ByIwiE1esWNone8E0/qT3m15z/pnn+U8sN2SmQcW+w1DHNFoYngr0ww8UtOvpFgAnrYNTKcgwHYaSrDHnCcdo42W0OTrzWzj+siZrK/5k9cv+QjtsZr6gaworQwlfwvIG++s43iDRXc7zKojjjI1o6d+XTQaPzmYIKtcFj4qls/LNJMx/IDhEsbXVIOEpWRx9J3C7l2SDrZm8BUX43HplHYqwODh90GwJZ1P8Qc5sVUA9Fvatj2mdg2KpqkUdWkmUuw7xHoqdvZHAk2r0GlSKSiRxLfifuCxw/8gQ7eMnZabmBt/V24DBtS+qk2Q7phxS3BIaCH1SDDZgGCU0RkcJgVlt0ETrpvoGk6SRFO9n38LLorHnNYJYl9PiKq41qG7BV02raUjS4TOc+r5K8oLUkl/wvEnrwSaleVEx0AEEh7GV/l9G1M/Mf5TBobs/rw1NF0Snq+iTNpLbtXJXLt+Kvpccdjje0cQLeTvq8mJqrJOgBcBubDNuoDk6g8PIN7Kg6SUhqghihqfPcTPdSgJtzER8lj+Ch5TOP7RDcE+PHHtdQ7LDiKX+Sqjj/GbgqWdzxbZmBkj0E4gusKdNixkEGmErZcnoM7LAy3tNE1uhrv7u8gA8GSke5KoGT93Wi+SKJdBn2OlLNNW8Ku5cPpMXxkC/ykFUUBlfwvGKvn7scSAI+9lIaIgxgmD7VhoVfZLLULinu+SUVcHjNrzezPcDPlpMQfyvFE2mQFsNvvPrb9lxQ/t5ajDcELgxW4d7eHly8Pa3LxseiSOz/9jKGrF7AvcwyH7WY2VX7JoITrMWsWtLhO1H/5G4RxYhRSuhDU4CMj+zbuiszgif1aY+I/TgZsVGz7DnVCsiytL1eW72ThO/9VyV9RWpBK/heIuioPXnsZdVF7QTMAQYwr2Ps+VbTL4LkSC/sCDiyG4I7UMae/YQg9ho88Y0KNujaTmtl7kX4DgLsqDMQ2N292s1PjEMR4JD+OimHKy08AT7B+yn1QoVPYsAuAnNirCeswGG/AhT//I8xuD26LmYNp6WR0mchlkb1IknV4dFvIxw08uo2vHF4O9LsFNn1AdMPpN5MVRWk+Kvm3sT15JSyetR0QNEQcPJb4gwYc2shX3fqhm078M1l0ya17nHyUsIEoj4V02YVHb/jzecdxfI5+58KDBGq8mGJsPDQyk1+dYe7+4bff3VhGKmzYRWHDLjSzQceRdUQ/4MR2eBipe29jcCACAN1cT7oucToE0e7QM3dcYT4E6V/yekQ2I5ybz/ucFEU5M5X829CevBK+fHsHUtcQgNkXhb0hDGfkYdAMssuLAMjr1It6u4MYl8G9uxvYF1vE5IOTGDb8f9THvdhs8YT3SzrrhVq+roz02pT7qKsoZ2eXFSwfOoZBO7fz4JwZjA6/jNl3Pcx38lyn9f4FENfQkdyaXuwPP8DieAsv//5fTLkpFnJubbZzVBQlSCX/b7J1Jix6GmqPQHQHGP1ksyWj1XP3I3VBculasgrmYfdWM/+mm0A7MTwyu7yI7PIiIgwbd/i7MjdlN6szB3NX2Wb8bhsT+6U3Syzf1oKCBUwrnUbJ4JJjC7fcTI+s4AXh+KeCnvu20nPfVgA2Z6cw9sEHmFbtPuN7aoYNgaBzQxYlUU6MtctYVVTB0P9DXQAUpZmFvqOoBG2dCfMfhtrDgAz+Pf/h4PZmUF/laZy+weGtRgBuhz10W83DM5cX8FzvsYzasYD0zG2Urc1qlji+rQUFC3hq1VMUNxQjkRQ3FPPUqqdYULAACH4qGPvgQ0QmJIIQRCYkMvbBh+gxfCRjdq7B6Qg9yZChBW84C2BAWRyvDruN6BWFHJz+ZGudmqK0G6rn/3UWPQ3+U3qqfndwezP0RM2OSrIK5mEy/I3bwlwuXOHhp7Wts4WxIi6Xm3a8Q+aqvcx5cBz77b2557yj+PambZyGJ+Bpss0T8DBt4zTGZY0Dznxz+aErr2DG+xvxa32xGCcuApIAG7LrWNl9LPU2BxFeN2ENDWh+qF+pZvdWlOamkv/XqT3y7bafpT15JczZ/Q7Z187A/mkdJz8lW+/XeHfwGOrtYY1TM/coO0hCQQH3bl6ORLKhZy6rxFWIaAOeimn2ctTJFhQsYNrGaZQ0lDSuy1vSUBKy7Zm2n6zH8JGMnLufPYUfI6Ouwm+OwtC8bMiuY1Gfzo03t+vtYbgtNpb/8N9cV1BK1Zy9xE3MbtZzU5T2TJV9vk50h2+3/SzsySvhw21v07vDGyysr6Mi6kTi/3LQUN66bhL1jnAQgnp7GF9160cdBhZnJQYSe1Iti6+6CYB0bxktUY467kzlnWhbdMj2KeGnryYWyqA7b0IOv429yavokPIPKpPWsrJ7RpNRTQABk4lXu4IWlkD96mKq5uw973NSFCVIJf+vM/pJsJwyN43FEdx+jlbP3U925gfYhI/VDWbeHSHwHMt5b0y4Ha+t6fh23WRmTv+xzBt+mM6DtrHjyuDat46Ah6kFr51oeLwc1YzOVN6RUmI3Nb03YTfZeaT/I2f1vuH9krhrWBYfBq5msa87JqFTbzt9DiCAErsdh7YETQhceWr9H0VpLir5f52cW2H83yE6AxDBv8f//bzKK6muz4nVgjNadiofSKbnafZl34PLHkNZXHzI76mNjCGn+lY2ZP6EOclj6OAp4fn8Pzeu4HWi4fmVo051pjKO0+fkqaFPkRqeikCQGp7KU0Ofaqz3n43wfkmkiSrmGcNY4s0m3Bt6FFBkfS2zDq3kUN0OpITikrnndC6KojSlav7fJOfWZqulN2wqY2TUO8yVMewrvJERBVdgMayUJQ+mLHkwkfV1OCNPL6lE1tdy2YDuZKcOYenCfN53TaGDVnH6Ac6jHBVKSngKxQ2n97ZTwlMYlzXu9GT/LYfF/tI0g6n69zlgJBC3txxvz/QmpR+z38fwvC+o0y2sq/wMj6HzyQeLeeHHE5rtHBWlvVI9/1bSsKmMyg93sXe3IGNGPWwbftpKWCM2lWA+aSlFCCbAzEP5vLMuiqmzt1FU4+bP+q245CmLq59nOSqUR/o/cvblnXMYFjsxvpBnza+TTjnVR008u+95OnhKQEqi6qq59qs5jc8JBKROfu0qBmxTN30VpTmonn8rKZu3nf37nyNlm4UoKYkYcfpY98uLE5GrC1jSP5n68BgiGmrIOrSCIat38FbHE736ecYw8MMvzTNJ0yrRWmi0z/Ge/amjfUKWd85lWOzoJ5k450dMNK8CYFt4OHkrP+Nve68k1ILDrkAtHw46gm3xu0wedcf5nJqitHsq+beSA+Fr6Jr8f4ib4pHuKuo1iPSc3i6rysrqqoex18QyZW4lEc4kpmffdFq7ecYw5vmGIYADT519rf3bClneCeVchsUeuyjoH/4Uk6gnwelnry2SSJOXusDpD7s12AOUW6t59tDz2Auiv9U9BkVRmlJln1Zw8M1XyK4cjOZIQAiBFhZPdIxASn+Tdj4TfNkvnZq0f9On5j5GbDfIKqvnSFwPYsMsId87LSb0KJlWd67DYnNu5cCqayj0fsQu/fscNHchIbkOv2jaL9E1gw3dqgHwan5+v/xPzRG1orRbKvm3AnkwDSFtfI6PydQxHCf/9bipsBdjGE4kEqcdPh4YRmGci7HbZjJh2VoAYj31PDupD78d3wuHpen0zg6LiV9c2y3UIZvFnE1FXPncYjo9toArn1vMnE1FZ258PsNijSOYDrxGVtmV9Kr9HVZTJKvih2EzRSGReCwBdE1y1ZYEbl6cTqeiMOplNbuWLzm/E1SUdkyVfVrYnrwSHN44tqKT5tX4gceOW4JPSBbbU+jl+jcRmhN7tZmR1TqRCYLkwk5cvm0pAJa0tCaTt/1lYT5Ha9ykxTj4xbXdWmxitzmbipg6extufwCAoho3U2dvAwh9zON1/XOYBC/ppz+h+DdPElk9BWf/1xkXvp6KJAfz04bgZy5XbY1HHOunRHjMXLUlHqvPxks79/OKWvBFUc6JkPLCnDdl4MCBcv369W0dxnnZ8Pe/smJDgIlpl1MQtwFv99mYw6rQXXGUbf0ONYdzWWptYLQvOOe9SXiQNTMYsWkdAMJuJ/WZp4keP77VY7/yucUU1Zw+9j49xsHKx0Y1+/Fq58+n7MW/kZd7O7dZ/oRFK2dIejcmLYlDC5zeR9HQWZj2CD+8O6fNZjZVlAuREGKDlHLgN7VTZZ8W8tKrU5hie483Rk6nLnkj+uXTsYRXIQRYwqtIHfQ2MRl5DPSHIZHUCoPBEa8z4tB+EAJzWlqbJX6AoyES/9dtP1/R48eTvXgRNaZY1shRGNJKg8WFFjh9JTMAAxNX+yTPzNrUIvEoyqVOJf8WsKBgAf+2raDB7gEBldmz0cxNx+9rZh8x/ecQJQXPx7hZ63DhPrKP5f260mPXTrIXL2qzxA9nvpHc0jeYx44fxtr6kcwVo0jRA1/bNj1tM7UOlfwV5Vyo5H8WFhQsYOyHY8l5K4exH45tnLf+TKZtnIa7Pgff/l9St+tZ/PbKkO3CrFVsvUxjkLmcIRHvssa4jLrKEE/utoFfXNut1W8wA3TNTWH87SMpq/4+P66uwWrSQ7az+3S6znuPD2Z8wH+mPN6iMSnKpUgl/2/wTQuXhHLoaDL+4kl4fXGAoNITG7JdhYhnyeUOiFyKs/ggAJHxCc1/EudgYr90np3Uh/QYB4Jgrf/ZSX2+tr7+rUYHfY2uuSn8YNoYRvksXJZ2BEMYTfZrhkG34iqia3Xsfj+DF89hxq/OblI5RVGC1Gifb3A2C5cA7Fq+hH0frqCruR++sBuRJ02/MHvvjdzTawY204lx/V5szORO6uw2uu7LJ86ZgNlqY/jtd7f8SZ2lif3Sz/pm6rceHXQW1lRmMz5yPW/kRDAgP5YItwm7X6dbcRXpNfWN7aoiJf/ssJh1s6fywqRnz+lYitLeqJ7/NzibhUt2LV/Czre/4IqwAG93/SfS33QlrrySQby143Yq3LEYCMpJ4A1+yCpxFYneauLqEpssdXgx+svC/MbEf5zbH+AvC/PP+T03Hw2jTsZyIMPKh6OKuG5rAaN2FTZJ/ADxTqiMgi9rP+V/G393zsdTlPZE9fy/wdfNbHnc8hnTMfe8guu6ZeKuXIow1yD1pqWevJJBrKnJxXt1auM2W8DLxCN5/HzG/JY7gVbSEqODIuMTWFF2ORPTPXzkcFEZpZPoPL2dAF5+WefdEQFeNz5gdFpfUlPUzJ+K8nVUz/8bnM3MlmtiUlnmdvH8M7/ng9+X8oMtnwBNn5+QgGE1YXL7ENIgxVPOuMMruSX30pifpiVGBw2//W621/q58kA44xsSeXdkROPCNycTQKITfvCJpOdugz17njnnYypKe6GS/zcYlzXuaxcuKS6ZC45IfpT/L+SvjlLyso+BP8njtt6zsdj9waRvN+HvFY0WZmLsZ9UkVVVy58yZ/HDANeTk5LTtCTaTlhgd1GP4SFb2rmRDkU78UjPZB5PZlZqEy2Im1KOJdh3uXCLR/dXnfExFaS9U2ecshJrZck9eCStmb8ddG8E1afXU3eFv/GkKYGzaV1ydtqaxtg9AnIUBu+v52W9+zJGbR1wyiR9O3NRt7ukn3J0szOhwgO99no1J91EcF4klYzC9Vn0csn1cPRQj2TtqNEk//UmbPiuhKBcylfzPwZ68EhZP30EgoBGZsYak3JnY8jQs80xE1Aoqo2DGCIEx2ODWuHdYRTD5ex1WnJrBkZtHcP0zr7TxWTS/bzM66Gw9csVUnlrxG3Z29tBrn5lMezcGJVyPx7EK6a46rX25I4anNozmiYffw/nRY3QBdQFQlBBU2eccrJqxnUBAEJmxhtRBb2N8bCdqhomo2uD0Y4lOeOATCWs1llSduEMZVV/D6tS1l2Tibynjssbx1LBnOJqjs3DYdfSJuxqzZqGu0yA8pqbTXHtMFt7seT1HnN35rdPBFzf52fvJU20TuKJc4Jol+QshrhNC5Ash9gkhHgux/14hRLkQYvOxP99vjuO2lQZ3cJWppJyP0Mw+4peA/3KD0md8HH3ZR+kzPoy+Ae5YKlndEPxwZfb7uGrdYrb2jWnL0C9K47LG8fnNn+N1GISZg2scr7eXMe3ymyl1xGAApY4YpvW9maUZA5B6DC4pmFFrZdHIOrZu3dq2J6AoF6DzLvsIIUzAy8AY4AiwTggxT0q585Sm70spHzrf47W1PXklWLRq/EYc5rBg2UH28eK8M4C0BdsE4qH2zgCR74DETFRdNSPyFpKtb2dzvFqE5Fy9POAqnPtKiAk4cAWcbOjQlaUdBzTu7+418WCtmSgpkKv/yo6klXycPQvrokWX1P0VRWkOzdHzHwzsk1IWSCl9wAzgkhxkvSevhMVv76JLwRy0gBfdFQdA7UTZmPiPkzZwficAUvDDd/5CL28hr9zwB6ZmpYZ4Z+Vs5OTkYB4ch8RPmCmKIdVrMBt+btJWkGd9mE+jbufBmP9yQ1SAidE2fuYexa27f8RndGrr0BXlgtMcyT8dOHzS10eObTvVZCHEViHEh0KIjGY4bqtb9cFeArok9eB6uue/S/WGMRi6FRlrhGwvY8BXNRiJ4O1JT/B8twwmp8S1ctSXlu4TBhIb9ha9wwfQ03WQx1z/5k+WN0jWKnAFrsIr78GqORBCEGYSjNd70T/g53+fLmrr0BXlgtIcyV+E2HbqMOz5QKaUMgf4Engr5BsJ8aAQYr0QYn15eXkzhNa8GuqDc/Os6it5+uZNPJs5jzdXRVF/bDGWU9X4bViLriHC5GP90F4q8TeTCGMuMf636VFYzC3ReThEcLpsp34PkqYP5JmF4AeHLSxZu5na+Rf/k9SK0lyaI/kfAU7uyXcAjp7cQEpZKaX0HvvydWAAIUgpX5NSDpRSDkxMTGyG0JpP7fz5CN9G8q2/56VrLVRECySC/A6SGcYwvDSt+/gNM4f2XcEVNXn0CdSf4V2VcxLdgT6ZB0jr6iXK7G3crMvQvzMmRxwPffAfiv/wx9aKUFEueM2R/NcB2UKITkIIK3A7MO/kBkKIkwvdNwG7muG4raZ2/nzW//lZPK6lrOhTSUA79mFHgM/iZnvlGv7rGUU5CY0Tt70V+D7OigwKB3aj111qvvlmdWyx+D6ZBzhqCyb8Pa7huI3QS5KW2jT8/QM0NISYGEhR2qnzHu0jpdSFEA8BCwET8B8p5Q4hxNPAeinlPOBhIcRNgA5UAfee73FbU+Vbi9iZnkp8ko7fcnp9X0gf26u2syz9Xyc2WiSdbPPZ0i2X6FH9WjHaduDYovD67Cc4YksgxVXBMuf9JFgM+oYJzOJEJVKXklXsxTk5l++sW8f8u+5h/P9CVh0VpV1plid8pZSfAJ+csu3Jk15PBaY2x7Fa0568Ego+2ktEYhey9FVsjh8OFIRsqwWartYV4XWzrFM/IkyhbwYr5ynnVryBEczZ+ib2o268RFHkl+AK0NNuwqGB24AdngC1lgLAJ9dBAAAgAElEQVQqt3VBk2tJ27aNF//2Oj/9yQNtfQaK0qbUE75nsCevhH/M2Eaq4WVb7SpcWSV4Yj4mygi9rqxhim98bQ7ohHnd7EtK4U/d1DDDlhLeLwl3/AD+3f+OxiEGRX7JF3U682p1vqjTKfIbBMxmkuvq+d6LL7Lvniyy9ixUD34p7Z5K/mfwz492YiqeT5yw8mlnCxsiLTxeU8rjldVYZNPasmaYMGw3gJRE1lWTWHaEinDJ9xPq1AifFva3G6+honNvhPSE3C9FcISW125n+O5IHst5FF8vwZFPX2zNMBXlgqMmdgthVkkVHw214x57HwsO1mAxCvhncTUOKRnX4AJgWmwMJWYTcT7ouSuWzkeXIVnGgpETqUzezuudx5w2E6jSMt694gZe+PPTOBKvQIoTv9ISg/rI/QA43G6iagvJis3iT1n3s2z1hbNcpqK0BZX8TzGrpIpH8w9znWslj+94nXRvGSUmjZTAiXLPuAZX40VASnhvezzFxzr4D02cxOSUi3rqoovSh33688D+DegRvfERhqF5aYg4gDesHJOuc7TWTlVCBVfv1Hi1YzYO4WXhu29x7R33tHXoitImVNnnFM8WFHNd8ee8sOcvZHhL0ZCkBULX+QH8LhPdSoJz/Jhj41WZp438amI/TNesYESMje5yLa6IpXgdZVjcbig9SkbNRrIOrKTStY3UhuADhDs/mQNbZ7Zx5IrSNlTyP0WR18/jB14nzPA22a4Bp47bcQsz5Vsicfh1/JqZa++8t7XCVE4xsV866R3zSTP9gy6xV/K2dgUrK2IwHconzFmGAKIC9aRWbOOud17n8LoYDL+fhTM/UxcApV1Syf8Udr+fdG9ZyH0COGoyYQBHrDH8vOuvcBaG4bWakL5Uegwf2aqxKk05bKmYIpYzrbuXBqwMrc7DIvUmbSwygM1XR2FxLKlVdaxx2mDR020UsaK0HZX8T6EZGvWmsCbb5uhDudIzjSzPO1wReJcO2Z8wcMhcVlr6IUwGKTn1XF+0uY0iVo7L6vwoBVkxDHW/ilXTiTzDtBoes5lYr5c+R8pwlBWxqzD0SCFFuZSpG77HLP7vHLZ8MZP7bw7nUJaJnvvBZAQT/1T9AdzH5u7RPAEsO2pB17ln2YckD6ilttzBv2+5m5fa+Bzau9SUCXA15Kx4Al9YDL7DFmw+f+P+nV1yWHnlNdTYY3iTCm6re4fvHficDXs60aMN41aUtqB6/gQT/6ZP/4uhOxkStpiyVCu7siM4akvgz/ptjYn/OGFIHHucfLfzl5TWxvDEiJ/zYf8hbRS9crLUlAl0u3k7I6tdZOSWIMzBOzU7u+Tw+cgJ1DjiQGhUiCRej/w/pvcaS27P/cyYrS7dSvvSrpP/ruVLmPbA3Wz89A1AxxEJhjn4AFdpsp3+uR9QROiZIj0BjQXRwxjxkxksGjyMdJslZDulbUwd0p/UHmVkXFWMXfezPHcMflPTi7hP2JlpupNDWQ6u3P135mwqaqNoFaX1tdvkv2v5Ej7950vozqrGBQn89QFK809M05BABdJuCvn9MaZ6NmrZADg0oVbousB0zU3BbCQRl+2k36AC6iJCr51cQQIem0a6r5y/LMxv5SgVpe202+S/+K3XkLqvyTZdmji0Jh1vINiLv5V3oIsdqTVdr8YsdLrFVfJR9xvpYLOoFbouUN36PIahW3EPNkj26CHbJFCB4bFQYk3E6jzMiy++qOb9UdqFdpn8dy1fgqeuLuQ+q9fPiqLBVLhjGSpXcFvCB9DNgXHsE4BNCKxmM+///EcUj+yrVui6gKWmTKBg6934G+L40R4/tlMm5bNKD7ca71O4Mon3tg3kuvIS1vh28dKXL6kLgHLJa5fJf/mM6WfcZ4nw08e9k/0zOrL5tR6ETa+k69rNJOnlvL3ol3TyCLQ+Ca0YrXI+wnJuYe8nf+DaAxX8epuPJJ8bISUJsowHff/jxi0Z1OyPBqMOs3M1V+T3oSoQ4LXlr7V16IrSotrlUM+6yoqQ203CRH/7JBL3d8Bs+YpC3y6iAvWMrlxKnFkyp3cu2zvbCCTaQn6/cuH53tgu/LF6O4fmHeE6SxjXl9iAesCBbkxiXcWnJ7XWMdyrGHLou8zs+/s2ilhRWke76/nvWr4EgQApMesBzLoOUuLAxqD468mM7EW4OZpBCdfTMTw4+tsiA1DhYkunXPRecWpkz0Xm8dsmYuSsY3ZNPn53NVJKGvy1rKv4lMKGU1YUNeqw6+G4ze62CVZRWkm76vnvWr6Eha/8DSkNEALdbEIzDC4vLCO9zoet7xCIDLY1axZyYq9uTA51ug09xYYANbLnIjTiV29x768+5kUhgDruq3iZCE/wPk5M51rScsuwROj4662Ub89DqxjKnrwSuuamtG3gitJC2lXPf9mb/yJwyk0/Q9PIT42DgA/fzo+a7AszRzW+Djdb2Z3dFwnqBu9FKtV+4vWGrlVIzMR0rqXj1cVYI3WEAGukj7RB/yXHnMjiWdvbLlhFaWHtKvnXN4Qe4eOxBD8ASXdVk+0u3QmASWiQMBqADqrkc9H61cR+2LXgQ3xlcV7yO9lJyy1HszRdmU2YAzzQ7X/cYf2hmvFTuWS1m+RfXDKXMIs35D67/9gYcEt44zbd8LG1ailhpijsyWN5auIw9TDXRW5iv3Seu6Uf6TEO+uxNZF32TqwRvpBt/VaNZcnJyI+mqAuAcklqN8k/f+fv6ZToRYqmD2xphkG34mM9/mO7pBHAv2E6vVd/TI/dB/ndxGHqYa5LxMR+6ax8bBRjO1QwZHscNu+pqzQE2bwGKyKymesfiPHxY60cpaK0vEv6hu+u5UtYPmM6dZUVmG2x6HEdCTc3YDRU4bGYsft1uhVXkV5zbOpfX0Pwb6GhF63F5DBIctdQPLJv252E0iI6bWwg/coexBUUUtpNYphOdAq0gCTugORj2z4WdKnlFd3GIwUL1JrMyiXlkk3+u5Yv4fPX/oHuC5Z6dI8VjpaQdbicjlXOkN8jHMFevXRXIUwSX6zAhirzXIryYm/itS63UrbkI37OfzmSZcNj07B7DToecPOKEYGw1AJQbDHz5NJf4fpiJrf84K02jlxRmsclW/ZZPmN6Y+I/2b7k0BN8YbJi7fkdZMCPb/9sDvdMxFwVRtJPf9LCkSpt4d8Tb8dntvKFHMwLR+/lsjUmRi6rpGeezmt6DB9HNV3Qx2cSvCzXU/vyE20UsaI0r0u2519XUR5y+/GRPSeTgL3v92jolMu/sg26lli4vshK0jNPEz1+fAtHqrSFYi34e2DPMLOgcAjzfMO42bqFCM3HnOhZIb+n0q7xv5UlTGnNQBWlhVyyyd8aoeOrP/30Gkf2nMQVFsZbPUqR2iL+uv3vWN+uOq2NcmlJt1k44vVT3CODVA7TcFgnXARH/jh0B27L6U/4OgIOZg3tTkpJlbrxr1z0LtmyT8qgUoxTrm2GMMgqbZrYpdnEG9e4+aTDZ6xwZlK5Oxzl0jc1KxXHsam6i3tk4Bx7GQ2W4JxNKa4UkHBDXT0LC4vYcqCQhYVF3FolGFBUydT1O6mdP78tw1eU83bJJv+9jj4sSrgar9WKRFJv11meU8ncYQ2UR4EBVEZD9Xd9uAcZSH809x+ajyPe/43vrVz8JqfE8Xy3DDrYLAgpSXRWEFFZAUaAkrASbqiv56nKatICATQgLRBgiiufXP0AzrBw1v/5WXUBUC5ql2zZZ1bBWO69bDO9Lt/OOh/MKbehm+BAuomVvcAiJLfF+hgYHuBG3WCZexKjE18gppOarrm9mJwS16R8s2HV8+zZFcWsLDePVNfikE2f/HVIyVj5FUMZyta0GLRXXmKkuiekXKQu2Z7/DTUBenVfiLQEGBgeYMjWOGJNBiCJNRmNiR8g1iSJSe5FZnIARj/ZtoErbaZ67F46T7CQoMeREgiEbBOJm79O+zuGK4IdUSJkG0W5GFyyPf8rHSDDXZSVZnLwYD+SosO57pCLTlkbSUo+2KRtdcDEr49+COP/Djm3tk3ASpurDghE2hoKGn5GdeE24rXTLwBCQHyvakbP28WiTt3bIEpFaR6XbM/fb/+UioJM9u4dgtcbAULg84ezd+8QykozG9sZfkFp5CQm3/e6Svzt3IKqaHwGxJiuoHRTPKdUfRqFWwOs7RrgitJDrRugojSjSzb5D2cFBw/1wzBOGfFjmDlQ0B8pwVdnpmhJIg8Nea6NolQuJJ2SfsKMSgdZ5RXIvWYC3tBlnRKTifdGaMQ66/jrT3+n1vtVLkqXbNknmjrcptDDNn2+MLa81gMJ9D1U2rqBKRes3476HncujmZdIJGyuARiNrpIHVyLZj7xESCgC6YlRFMZEXw4MHL/XObPD/43ysnJaaPIFeXbu2R7/kflOCKkPeQ+4Q8+zOO12Ohw6V7/lHOQb++MbtZ4Y8JtlJXEULw2Gl+DKfhJscHEfwPRfBIZQbwz+J+nX7GPg/oWFi1a1NahK8q30iyZTwhxHTANMAFvSCmfO2W/DZgODAAqgduklAeb49hnUmPcx8BANcstuwmIk6btNQJYy4vwmyyESwepTzzekmEoF5kib/A5j0WDhwHwwEfTqZkfRmUUvDtCsLK7ie7F/RldMJnFI8JxCMF3awJs0da0ZdiK8q2dd/IXQpiAl4ExwBFgnRBinpRy50nN7geqpZRdhBC3A38CbjvfY3+dSL+daFLBD+vNBdQLD5rux1J2GJ/uJyYsgXsn3KTm7lGaOD7tAwQvAIsGD8NWv5LI6vfx2rvji7qL5RmRbO1tMHKrmz6FPva4TQysupKGTWWE90tq4zNQlLPTHD3/wcA+KWUBgBBiBjABODn5TwCeOvb6Q+AfQggh5ZnGU5y/cqtBsk+ji5FKF9+JaZlLUw1WDrDy8IQhLXVo5SI2NSuVR/MP4zZOqvM7crnCtZOFCd/DL4KlxNpwEwsGBe8p9Sn0sdcDYR9t5WBqfzXvj3JRaI6afzpw+KSvjxzbFrKNlFIHaoH4Zjj2GR2uXI77lLNza3CgYpVK/MoZNZn2AUgO1PL/it9ideKExsR/nN8sWJLjAMAtIdFn5W+fTmdWiZoYULnwNUfyDzUe7tQe/dm0QQjxoBBivRBifXl56CmZz9bInm62V35FqdXAAEqtBtsrv2JMr/rzel/l0jc5JY71Q3tRPLIvW665mqsiPqeS0NN+1IYF/wtZHNUcSVtCwHiP6Z/sac1wFeWcNEfZ5wiQcdLXHYCjZ2hzRAhhBqKB07pHUsrXgNcABg4ceF4loegpf2DUy09Q9uYPaaiXREUIRt03iegpfzift1XaoyidBCqo4PR6frTLQJi8JOR8SF2HTfSuGMxl6zy89eVb3HPNPW0QrHIxOTTzu+yJzENYBdUBwbLKMNKrbuexKS0/EEWcb9n9WDLfA4wGioB1wB1Syh0ntZkC9JFS/vDYDd9JUsqvfZx24MCBcv369ecVm6I0hw8W5bJLdOcN/g/fSaUfS0Dnfv/rDLN+ie6Kw7l9NGHWwxzcez+lKUvoMqyLugAoIc0qqSIw7x5MRoBdth7MSrmWElsCyb5KetSt5fIycc4XACHEBinlwG9qd95ln2M1/IeAhcAuYKaUcocQ4mkhxE3Hmv0biBdC7AN+Bjx2vsdVlNbynriTK/RVfJ9XSZBlIA3iZRkPiH8w3PYlQoAlvIrYAXOp9XYlK34p7+Vez8f7G9o6dOUCNKukis8W72fAoR9Q6fk1b2TcSbE9CSk0SmyJrIkdzdr4lp80sFnG+UspPwE+OWXbkye99gC3NMexFKW1FdpHM/fgIcaFL2BoxAr89WaqXBkUHuzLcu9d2GwNZGZuIin5IPF9Pqb4s0cZ41nI/M7jmaVW/VJOsWLZQX653YPNiOaVrnY8pqZ9cLfJTkHMN3bcz5t6vFVRvsHUrFR+Un87Ow72YfTsTzBb7fgzOjbOG+X1RrB3b3AEWWLSQVyBeMbZPmJbxUCeLXCo5K80ccdOF45jz52W2kP38EutLToYEriEp3dQlOYyOSWOZwNOStIzeOWex6jL6BJywsCDB/uhu+JwmKqx2xrIPbCz8YlhRQF44eW/k+w5MeNAsif0PddkX2WLx6KSv6KchbuuH02e4eK5xQ9jNUIv9OLxRPCrVVNZnNKA8+MEIrxuHIavlSNVLlQ/e/Vf/L1LLqX2E2l3yh4v9sApK8YFPHRsONLi8ajkryhnKXr8eO75/Yoz7m+QVmqljRWuaJ5K685e53YidtTzyKK5rRilcsHZOpOG53vydNEnfLG8ihSPRB57zOn6Ep0ntntIcQcQUpLiLadX3R6SKlr+E6NK/oryLfXsmcCpK73oUmODfuzBdmmlrnYMu3rlk276H3MLo9Wc/+3V1pkYcx+Cmi5UBh4hTI9AAAKBlBIpJdcWlDLzxWm8s/J7ZJk3csC7loeL/9Pioankryjf0q23/piO5grMeJES6g0rK/2XccA48RSw1GPQzdDAZn608XVufe8AczYVtWHUSlvwffprtIAXp34PJmlpsk8IgXRX0fD5VAKH15E0R+cBXmOg9iU57Grx2FTyV5Rz8P9+8zJJYbXMdvXnQ9/lTRI/gDDXAFAd7qffdgu3lr7O1Nnb1AWgHfntwgVYXMHFogJnmB5EOE6MBNMawKrB+OgACFOLx6eSv6KcI++BcVzlsWMxTqnPCh99fJ/y8ss67z+nE1+5nXEbj+B3rOcvC/PbJlilVf124QKWLSuh6Nj8lYasC9lOuk+fBDDGbFCdM7JF4wOV/BXlnGmGnZ5+M9dUriBcNgASYa4mJ/Ahv/tkI4nO4IyGDl8D4V4v9x6YRQf7MlX/bwfeWenkfssS/uy/DQ9mdroDnDruS+pevDs+avzaOLbqbG0AdiaVtXiMKvkryjkKDwve9O1Sv5vZehhTovNIynyJR5duxK43bWuScF2en/t6vseqVX8LXgC2zoQXe8NTMcG/t85sg7NQWsJ1lh2Ehy2lLDXAP9I68LteL/Bi6n8oM1cjkRiuCjyb3kYvWguANEmctwTQDY0qXeDxFrd4jOoJX0U5R0Nv782Xb20HLZIwcxTCHc71R64nwfl+yPZhvgBbvJLeHdez6NN55Oj/AL87uLP2MMx/OPg652vnPFQucI8vzWeo9T2eSYpjpFjN/wxBQKtmqX09S2PWYxGSnx3wc7lbYAICsQLnTQFqe5oo8ulk2SV2W1qLx6mSv6Kco665KQAseHUwLt1Jvd0DgCssjHCX67T2whFH7q6rGOp7FY++DXA3beB3w6KnVfK/CNXOn0/Zi3/js/ROvJM1mYzMKPxCZ7nuJWBu+lCgXwpe62zmt78P/r6YXLGsebcDvYWFrMEb0TQHWZ0fbfGYVfJXlPPQNTeF5LIuzJm9gfBwGw3Cy9acHAatW4c5EKAoJoL81Dg8FjMOYadTUQbzu45iQvnnod+wtuWf7FSaV+38+RT/5kmkx8MbP3oM1vuo1XS6lA8gt/BGInyx1Furyev4MfsSNwBQHTgxp4/uqGbh1Q9xW3E0Oz359O//DKkpE1o8bpX8FeU8RY8fz9iNG3mtLhJTuJ/CzMsA6OYMI6PT9fSzbSTK/BZmUUGdvpx/u2/EjIEPM1ZOuTkQ3aENzkA5H2Uv/g3pCfbiE+siuM3ZgDPvBVKkwCKDQzYjfXFcXXA7APsSNxBrOvGQoMujUZ2Rg/lAPXvy72PcDS2f+EHd8FWUZpH629+SoHWhwp+Fy7BiTh9MepfJJNg2Emv5BxatAiEgyuJlinMOABaps5VuJ97E4oDRT57hCMqFSi8O3pxd3G0yN65roMhkEC30xsR/nMWwklt4IxYhGRcdHB4sJbzvv5wpe7w0CC+jR49utbhV8leUZtJnaCYeXTLTezlDAt0waxaizdPRhLdJO6sWrAE7A9EscA0BBERnwPi/q3r/RWZWSRXlsfEs7jYZI3UEFkNjuV0nMmAL2T7CF8tdER4Ghp34xHegqpwxJV64IpacnJzWCl2VfRSluVw1sTuetZvI87uxmyIBMImKkG0NCfdZnsKZLBh39f2t+p9eaR6PL83n3aUH6J77OOPcZrRjfWmnJqm3VhPpO30dh0itnLu2l7MrO4LSZDtOv5k6rYpE84uEF+yHrU+2WgdA9fwVpRldectohhmHKT02a2NAhn6s/6hMYKM7AVdDBx7dcbAVI1Saw5xNRbz7xX7wBBhJGWWJ6ylPXkZlQh49RSV1sXvQaDr5nxkPA2LfaUz8PsPE3DoBphjCzUtPDPdtpec9VPJXlGYU3i+JP911J1ujtuCRBrX63RiyaQnAJa38Wb8VMFHqrCUtkMzeUaOpnT+/bYJWvrXffbITApJOWgWuqEN0OryHG+fP5+YPp/P059O4YX89fR0mHMcG9dg1SX3nRRwYspuSJDuVATszqjU214cxyXXS78fx4b6tQJV9FKWZhfdLIrZiOvmLrqCndTT4HsJtfos0UclRGc878blc2eULbrLPpNITS1Z5X2yX/5zyV2YDwdFDyoVrVkkV1XU+vN2jOZQcx79sOSRVVfBAQOOadasId7mwbZrHtrSVeBPT6Rt7FZdF9qK2/Go+W7+bD0fVUW+rxhaIZ6Ju5XfF65oeoJWG+6rkrygtwJTxU7ShT1MQ8SkAj371O6q9seSmrOOenjOwmYKjPRIc1TgzlhBWk02U6XYq3/pYJf8L3LMFxXi7RyM7huMSwa59WXwiz9/5IADXrFuFORCgW0kVS+MiWVf5GQAdI3oSYfaxeshDkHMrDc/3JLz+9FleGyLSCG+F81BlH0VpATd1vxNL+IkRHZOz52HVfEzK/rgx8TcSUJE9C2G2YU4b0bqBKmetuGQuK1cO5znPBERHB4imi697bTbemHB749cOf/DfPyB1tlYvwxVwUmNOa7yh+8fM7+PSTikJajb+mPn9Fj6TIJX8FaWlmE703/rG7eLunu8Rb68O2dRvr+RzfIiwOFX/vwAVl8xl9+4n8HiPIpAYZ0idZXHxja/rbPbG166Ak601a0mqPjFb53/iR/Hzrr/gsC0ZA8FhWzI/7/oL/hM/quVO5CSq7KMoLaRX92fYseNnCAE+bxiDkzef2llsVOuJ5k94qBeS9T1H8vhvgg97qRLQhaFg//MYxom5mISUyBD/mElVlQB4TBbe7HED2XILAJrZzsrw2WQ4MhvbptssfJQ8ho+SxzR5jw62pit+tRTV81eUFpKaMoH09DtBQlRUBVW1Sfj8Ngyj6X+7QMBE0YEc0rQKXpFevNk6H3bsR9mLf2ujyJVTHZ9ieX2Did8eCiOi8p8I45SH93xe/t/cGZQ6YpjW92Y+u2wIAFJoGEkx+EQCkQmJje2nZqXi0JpeQByaYGpWagufTZBK/orSgnp0f5qevf6Kz2VjnyeNP1Q9R37+EDyecKQEjyecvXuuoKoikwHmIjwS/i/6MO9nj22cNkBpe3ZbKh9UWfhfpY1aDezu1URU/RvNXxGco8GtY+xq4MX0Sdx77a9ZmjGASL2eMFMU3rQOdOm8l67lcQy//e7G95ycEsfz3TLoYLMgCPb4n++WweSU0x8Oawmq7KMoLSw1ZQKp4yew7ompXBG+iwpfFhXlWae1Cxc+IiTk508gKf4gh3Ny6HHS/jmbivjLwnyO1rhJi3Hwi2u7MbFfeuudSDt2wHENKxs+CC7NdozdtRq7azVWj53a/U+gayfKNTbgp5ZEhl52D7s7/xFnsYNrf/hjegxvujzj5JS4Vkv2p1LJX1FaSWfjMHt9TUd37E1MJy+rF/U2B+FeNz22+vlrVT1aw/9v787Do6zuBY5/z7yzT5LJThYIEAg7kSWAgChLERUQharUWpd7rbVX76W21iu2elHbaitel7a3LXXfiggKAu4sAiLIKrJDwpqF7Msks70z5/4xWQiZsBhISDif5/GZmXfemfeXeeT3nvd3zntOEt9dlsnVtfst3pbL7Pe/w+0PzQuUW+5m9vvfAagTQCt4I+crGmX+k/gsHsYXr+brmMupMkbQSRj4GRZ+YJBo0f9gROYoku9+qnUDPguq7KMoreQ2yzJMgYaZHg8kpPJl78G4rHYQgmqrne1DoqhOtmHwGqjyeetv9X/m0331ib+O2x9QC8K3knxX8yU4h1ujd/VB7sz9Fy+5DrGQCEais7HgE1LlNoqsP+K5555jzpw5PPfccxfNGs6q5a8orcXZmUmlB/lEphEQGhvT+6Nrjf8J6kaBNd1KTb6V5PKV5K9/iWQgrzz8bT955e6w25XzK9ogGVVZzayyCpICAQo0jRdinHxqj2TIvmiCRjPexFTWOUtYx0qQkhTvtxztfx9Lly7F7w/d21FRUcHS2mG8bT2Zn2r5K0prmfAYQ8USpolPcZmtuCy2sLt5I6zMKX6FgevKWe26AlY8QUp0+H2b266cP4u35TIrV/JkSTkpgQAGICUQYE5JGde6qvGkplOdkYnubBjjL3QfBwddxqIDWn3ir+P3+1mxYkUr/xVNqeSvKK0l82Y86ffSj328dfkk8AbC7pbqLeS25JWkxVay35XBjgoHv57UG5up8eIgNpPGryf1DvsdSsv9fd6zPPiHp9m25J9c7S/BLION3rdJya+qStkav5WjjqP122UwyB5/DCtH30VFRUXY725ue2tSyV9RWpH9zqdZ6ogm0VuKtq8Si7fxWHFbwMPsnHloRkniZVVIjHzEVdwwOJWnpg8kNdqGAFKjbTw1faDq7L1A/j7vWY4VVBPh8yCAyGD48lqcP0jAEGBXzE6QEo9BY5WjLxsGXcHs9GScTmfYzzW3vTWpmr+itDLzjP/jjoVv82LNTfz0vcUs/uF15FoSSfUWMjtnHjOKQiUBkz10ZeCRodLODYNTVbJvBYsKSnmmy0iqe9qI8LoZkbML/xEDZnuwyb7+mlD7ucbo5uu4FMZ6tnMi8kqeHd+HGUmx7JgwoXpj+UkAACAASURBVFHNH8BkMrXqco3NUclfUVrZ5PTJHMx+k/6xezjgjWf9ylsxO5qWgPw1GtjhFu8o5r/2PjPvnN4G0V5aFhWU8qvdh/FY7fQ8cYwRh3YT4XUz1/IzJgdWcZm2t37foC4o3B4J/cFZI/hTznOU2m9m9kNX1e9T16m7YsUKKioqcDqdTJgwoc07e0Elf0VpE1ff9jP4+59ZljmDI1/G0WNIEQZjw8pPQV1wYkck1hEWIrEzfL+FD3/7R67/3X+3YdQd1561q1g7/w3mTrodT2QMPU8c46r92zEFQydln2ZlaWAiAY9gsGUP/hqNom8jqTxqQ0oTD7mq6Wz/MfF606SemZl5UST7U6nkryhtoO5OT/+bH7Ok63R+uOU9UgaUY7IH8NdoFH4byZHoG7mxJhOpSXRfDcl6Ir/6x2959me/a+PoO5Y9a1fx2by/oPu8VEZEAzDi0O76xF9H14x84RmD7d2Gztoqm8CbN50hlVdRo3uJHVPTqrG3RIs6fIUQsUKIz4UQB2ofY5rZLyCE2F7734ctOaaidBR9x4xjSt9M8mM6kev/AQeXJbHn3RQOLEvicMwMUtOn4TBGIYTAbowkwdyV/ocFj698s61D71DWzn8D3RfqeLd5Qsk7whu+g7fGbq9/7tOMbB3Sn9iKIQRrSrAm7sUx7foLH/B50tKW/8PACinl00KIh2tfh7sudUspB7XwWIrS4fR54DZ+dNcE1vSZjaVnaCEQj/UE11njMJ3yz9NoMDFMjmHr9g/5beUgfnfDwLYIucOpKimufy6RWAISl8VGZJgTgNXtIYigODqWrQOv4eXhn3D/p8tw3nEtedH9+eC+u6gqKSYyLp4xM29vMpfPxaSlyX8aMLb2+evAasInf0VRmuH+USHWpS6CRCAwUB1xmAh/atipZJzY8MXN5K0NR8nqGqtG/7TQooJSqiOcOKrKAfBYHdyycTMbU3tz1eHvGpV+dGngU2M/IsdOQhq8rEl/F2Gq4N6F/9eodARQVVzEZ/P+gjFXEHHYTqDcixZtIWpSNxyDE9vkbz1VS8f5d5JS5gPUPjb3V1mFEJuFEBuEEDc092VCiHtq99tcVFTUwtAUpX3wGd0kDXkXQWgoodEXhTsow+57wipY1tnBE+4C/rh0d2uG2eEsKijlwX3HWJ1xJX5hJDvZxeC9f6DL9iVU7tJZ70nDFTQjJbiCZr7yd6UgGIXXWsKK9Hc5mLCFaGuoj+Dk0lGdgRFjsG4zECgPbQ+Ueyl//wDV2wqbxNIWztjyF0J8ASSFees353CcNCllnhAiHVgphPhOSpl96k5SynnAPICsrKzw//crSgdjtSRD128AKNl6E3ZXd3ZrQQbZBcaTVovSpeQvGRaCQqL1XYft6FgOjJ9A4gO/6LArfl3IaayfysnHe9xFdkVnNvVKpx8aiSfsVGXoxNjWcsg1lkO++IYPCB9R8Yt4NSG0OpcEHh7+MNC4dASQ5uhLRtQQxCmrfUl/kMpPD18Urf8zJn8p5Q+ae08IcUIIkSylzBdCJANhT2lSyrzaxxwhxGpgMNAk+SvKpSi9x4Ps3fsbnF2/4dqCT3j5yEJygxJqAvSzatgMUBP0s6N0NZlLtpIVCGJLP8HgATlkb7YS6KBLPl7oaaxzvX56783mctMRrETWL8gujCbG6JH4o1eRXzUcqUcjjOWYEz9FRjUkfmkZy+T0yQBExsVTVdxQrciMuapJ4q9TdyXQ1lpa8/8QuAN4uvZxyak71I4AqpFSeoUQ8cBo4E8tPK6idBjJSdOA0DqxVm8xLkOQyKBGrl+S69fRvXvQaz4HdBACn1HjyOEkhssCdlw2nK5Ll1H43PMdLvmfbhrr85H8Bx3I4ZZgDSP8I4jAigsPm4zZ5BhPYJRGRvssfJLxx0afiRaSEkM8nsgZjDzesED7mJm3N6r5241RzR5Xi7Y0+15ramnN/2lgohDiADCx9jVCiCwhxEu1+/QFNgshvgVWAU9LKVWxUlFOkpw0jdGj16JbU+hi/wg/DVVP3bMO0BvtHzQYKMzpRLWs4Itho9Dz8qionSq4o2huuurzMY314m25TD9SzFWBPkRiQyCIxMYYvS/peicA7AF7o89IYWJytA9f3B8YedxKls9V/17fMeO4+p77Q2v0CoFbVjd77KhJ3Voc//nQopa/lLIEaDJJhZRyM3B37fP1gBqTpihnwXTd48z44H4er7mWxEAQEyYIVoXd12c08I7xWiZkbABGMbGDlX9Som3khkn0LZ3GevG2XD54dzUP0w0TjWdKNaExTO+Bw7iaXb40AlochkAJQS2OaucP6a29wG0bPwvNz3PK79x3zLj6oZ3V2wopf/8A0t94PiD75UkXRb0f1KyeinJxybyZb8c/TYaowCzM9P72YaynzAdfx6wH8WPma/dQlsYOYEVcXwqfe76VAz6NHQvguQEwJzr0WLsq2dm6UNNYf7hwJV2t2URgDft+BFam8DlTD6+kNPV5itPepDT1eSIcvVjs0nA6nUydOvW0UzY4BicSPT2jvsSjRVuIuaU3sTdktCj280lN76AoF5kRY/6Np7/6krFH3SSJKkQefNclgaChoa1mCAY5aA4lkoqggwFmDy8Ovgm2L+RCpJflOct5YesLFFQXkORIYtaQWfWdnWHtWACL/wOCtSeuimOh1wCZN5/VMevq+udztM/ynOWkmI5jq4mjBonD0LRTNkgZFgL0Tsur32aSHjoVbOQoybz+wANndSzH4MSLppUfjkr+inIRumNST1774jeMTPaT/F4VHIM9KfH4jAbMeijxL+k2CQCrQTDQncHGpF28PnAK953nWJbnLGfO+jl4Ah4A8qvzmbN+DkDzJ4CP/7sh8dcJ+kPbzzL5Q8unsa6bsK3urtu3hmdzTc0tRFb2Yo+p6XBaiY9448sAWK1ukBI8AeSBSk6IbP405cHvHcvFRpV9FOUidMPgVIpTjxETXUPlTJ1OhioMFTH8o+vdPNvr/vrELw2C6r5RHE7qhCPxIwotzY8y+b5e2PpCfeKv4wl4eGHrC81/yF16btsvgLq7bquKi0BKKouLiNybhr06HUFoNNX2mgA1AYmUEndQJ9b4Ag7jlwAUaBoRG5ZhW78Xh/Y+f5py0+mvdtoZ1fJXlIvU7Mt/yY65jzGsfy5bb+1EcX5X/BnRBA97EZ4A0qqhZ0QSTHGwIs6PtaiKXu6D5z2OguqCc9p+sTj5rtvdPTNZefl19Ni3EGOZmU4nviE950Os3jJclhh2pF/PiU5Z3JcUSvxuIXghJgpz9Arcqf/LH/pex+Sk2Lb8c847lfwV5SI1OX0yj1hWULJ3P4FSH7KnGXdaNJa49Tgq3kPTS7DXxHD5F1NIdA9nd+pgxnRZAsw6r3EkOZLIr84Pu71ZttjwrXxb6yXQurtud/fMZOWwTvQ69iLDXb2piF1LcSdJ2ZgudO9egog6QVzpS9g2f0cQqMHJk3EaH0VGYAiUMLdvN2YkxYb6MVY8ARXHwdkZJjx2TiWsi40q+yjKRezOmQ/hqZpItW5G+H04y1cTWfYKWqAEBNSYy1jTYz5Ftm+4PPtWnJHJLNvw2vc61p61q5h33108O3Mq8+67iz1rVwEwa8gsrFrjkTFWzcqsIac5yVz7R9DMjbdp5tD2VhIZF8/uHpl8NG4q7qgr0SNvAoz4zZCQeIiuA7chnDoICMZJrOM2sCf6EQp8z/NRZAQAKY7khsS/9L9CHdfI0OPS/zrnEUwXE5X8FeUi1mtEEldenoEmHJiLcnFUvIeQvkb76JqfjWnLMATNVGz6Iel3/pED4yec001fp9bH62al3LN2FZPTJzNn1BySHckIBMmOZOaMmnP6+nfmzTDtr+DsAojQ47S/tmpL2TtkLJ+Mm4bUbCAEg47noCFJSMihV+/1aNopS2ea4UTPhSQZXqvfVH+CW/EE+E+558DvDm1vp1TZR1EucoPuvY6afR+z5sQx/CL8DV8uSxkAAS2Ko2lpdD16lPxzuOkr3KyUus/L2vlv0HfMOCanTz73zs7Mm9u0LPKXxM4EjKFx9jee+JxIbw0JCYfI6LUBgyH8vJFBWymVlmwE/Ykyy4a/ueJ4+IM0t70dUC1/RWkHRj33Z/K798fmNYV936bbKInfiN94iB2XhW4+kh7PWd/0dfKslGmOvkzpfC83d3uIq2wzWnUK4oqlSzkwfgJ7+vZrcvWyf2MBrz/yFX+9dyWvP/IV+zeG73BeVFBK1vpdVDhC/Qs3nvicZ/c/g5MqunXf3rTFfxK9JpZF2jRKUv6H2SNmN7zh7Bx2f4/VSH5BkynNmlUXW/Kq7WSt38WigtYb/XQq1fJXlHYiqfAYcWkTOBZcgTA0jKHXghoDygYQNHqpjD1K0GAnIAxoMog/L+8039igblbKNEdfhsVfi9EQOsnonXexOe8J9LJSjMZokBI9UIHVkkx6jwfrJ6U7HyqWLiX/0ceQntCwUj0vr/7q5UTiMFa9vRfdF5ouwVXqZdXbe4FQaazOI6v38c7qQ0hPAMMEC0FjPI8c+if2oJcJrKPQ0vycO7pupmDXdfzh8sn8e7yr8ZXOhMdCNf6TSj8BAxzsaqFob2h2+zP9FnXrB9St1XDc6+fBfccAQv0KrUy1/BWlnYgzaySUCDz5NxJTKUCCzW9jSPEQ0qrTAAhqBoSUaDKIx2RGt4W/UjjVmJm3YzRbyIy5qj7xVySt50T/19BtJYBE18vQA+WAxOPNY+/e35xTq/dMCp97vj7x16m7evl6SXZ94q+j+4J8vaRhZvjF23J55/Ns8ATIMq8gtmg5BL2kekNXLpnsQ/OGT3nBoIHiTTP5IHkSf+nXk99ljm+8Q+bNMPVFPBYjEnBbDOzJiOBEJyvBoJuc7Lln/PueyslvskiPOyh5KqfpSKrWoFr+itJOjJl5O66/vcD69NtJ3yXp1jNQPwe9saIEQ001/rhOSJOZxdOmMmjbt+QkWcjcseCMtfe6Ccnsyxrm0inOWITUfM19pD7pna/Wv54fPgl+ktqdv4yyUmG346wJMm6HG2v1ejamLcNlKePthcnMGjKLZz61cb1cy6Pmt4g3VDLnRCLvGF3kWuLo4g2VtfocKmdvr0iCWsNdvb6AiarNNzCoagvPa8uw1swBmvZv7KlIIHd4NOGm6Xd78ph3312nXbc31xt+jqbmtl9oquWvKO1E3zHjuPbns5hYsol1nS9D1xsSv+aqxNcpFWm2gBB4bXY2jRjG5sShVC18pNn6+Knfb4xpGNKpW0tOu/9XXMHPvI+et/q1MTm5ybYvho1i7m33UOHQQAgqHBqf9f+WVT3fxWUtA9Ew3cTQys952vQS5qN+Dn6YyC3/MPLy3HV8Vqnjrs3YyUU++uyvwuwJIiWUuKM5XNqFG73zePWyy0575/La+W/gd4VvL/tdxkYjpMJJtYS/Cmtu+4Wmkr+itCN9x4zjb397itFV37AlkIIuDVgKc/ElpICh8QyYQc1IfKyZBT2HsOK1nWd1Aoia1A1hMnDQkI/H62h2v6+4gpf4OcUiEUlD/bolJ4DEB36BsFqJSquhx9QT9Lkljxl9VnNd2ZeN9rNWvcfIPW4ef83KvYtTuHN5GjM+iuUh0wL8RzTyNznRa4wIIKEShiy1847HSbHJEFqBq8jBe+tm8t9r57C/pCc3FG/nv/rM5q1ONwLN37lcVVJM3sZEgv7GTf+gX5C3MTSBW90IqXBmpydjO2UiOWtA8vNd7jZZ11eVfRSlHRp/3UiK33wJX1QUQvchTWaOOo6yM2YnbqMbm25jQNkA0lxdWJM3kv4T32P9/Mado+HUzUK5eclabIcGkdFrQ9jRMQv4MT7R+Mavuvr19+28dE6diqnkK6zZ/8Cgher7Kf5int3/DAAfdJoIwJU7Cpm2zsHe1IaZTk0BEymimOwdichA4zatVYeMFXbG3RfFELufKU6d67UFXOcyctnxUvTKGD7oNxGDHioNNXfncmRcPOXZoZp9yohCTBE6fpeRvI2JlGc76/c7dT3fOnW/y+/35pIf1Onkkdy338ukAp3yowcAWnUWUJX8FaUdumPKFOYddVG1dgnSEMkxaw5b478jYAglarfJzfbETfTqvY67fAGOH+lF4pjHgWaX5K7nGJyIa4kHV1E6AN26b8diqUb3W5BIzGY/JSIh7GdbWr+2ly0DrXHHrj/HwKwlL3N/9SsUxsZjd8PmbrGNprgGKNCM6DWNr37qxFWGHne7NBL3mYjfmc6YuNE4O83lV71+CkEvjvIFp71zuW6pxvJsZ6Nkf6rIuPhm35uRFMuo1w42Wsf3K+Me9hrykEs+R3woGDp0KFOmTGn2O84XlfwVpZ265z9m8kKhG5trBztjv6xP/HX8wEeVJoalBEiJ2E/MIT/5BUvOqoPW6XRSUVFBUVE6RbUngbrtDzzwAKnrd3E8TKJvcf36lJumKg7byN/kRAYEBiCptBgJeExNU9cLMVH8uz1AoKbpeyVRkOzXmVVWzmh3kJvj7uLHkVn8qteDLIvPIrL0FbqLw8w6zZ3LdR25a+e/QVVxESY9gG4QyJNOQppmZMzM20/7J56a+PdoeVBbDZJSsnnzZoALfgJQyV9R2rFx04dy/Mj/UVMSvsVdFghlFWmG6jSNfdvnkHzNmZP/hAkTWLp0Kf6TVhEzmUxMmBBatXV2enKjMesANoNgdnrTTttz4uxcO39OSOGOyCZlHAEYAwF0Y2jY5YG+WURYrEQUD+XAgBLSt6yBQEPcQgtyWb8KPjseGqMfNApckQmA4G8zH+JvAIQfoXOquqUaD4yfgJ6XR250BPuSY/GYjFj9Ov08vmZH+9TRoi31J4C9JyX+k23ZsuWCJ3/V4aso7VhmZiZmeykxxmDY9w3A5upQKcRrFawrzuDqhVezPGf5Gb936tSpOJ2h8sapSxfOSIplbu8udLaYEEBni4m5vbu0/GalCY+BqWGN3nBlnNzoCAIGA/scPdmaMZEkCZFeDz30JPqnzMAy6HaELRaQGO06ycMqcHZruDkrX8bxH5QgbOFLRGejblhqarmL8XuOct2OHMbvOUrS4TNP91DXqQ4QfpKJ0BXAhaZa/orSzmllgslOP++WmfHLU0aiIHi3zAz4GGXQyP5iAObRyWdeiYvQCeB069TOSIo9/3em1t2PUDt1sjFCoLsa77IvOZa9kb3I6ZTJaO0IonYY5zC9B0aM0GUE5i4jsBlWEWP6CwbRUGZxSzPZ+tUMtnxN9PW3fu8wjcnJ6GHung43XPVUdZ26lZ8eRrjDnwBEuJsJzjPV8leUdi5qsYFBliC3xPgwhEklfilYXmEmdv+t3LNzCd91z6I4cMPpV+JqS5k3wwM7YU45if/zJ1Z1H84dVz/CddOe4W8DrsdjMvJ1zOVcZT5I1yNHmfLhUm6e/y7i08fxHdtY/zXu4DjK/PejBxOQEqr8VjyyB3bH+/QYOrFFI2vqhqWeTFitJD7wi7P6vGNwIskPD2fosKyw7w8dOvR7x3a2VPJXlHYuKq8LJSsMjJR+whd/QrX/H/eezKa+g1i4ezkTuPKiX4kLYFXnIay/pju/mvQCb0U/wM/2fUiONZ0Hbe+ReiSPYZs24aipQQDSXYp3+5tNTgA7xMtU6RHMO5iFFiwi1zUV6/V3tygu59SpJD/5BMaUFBACY0oKyU8+cVYzqJ5sypQpZGVl1bf0hRBkZWWp0T6KopxZ4gO/YMijj5GqHSG6n53yQNOSQUCLo8hm4Lk7fkpC9lx+vz/A++mj2yDaMzt50XVbLw+3XXEUk1EnaqmJ1YmDMfSP5ifuN9i+Ix1j4JR7EAI+fLs/wNxlBABuA7yYYeOmL1PxWq381fdLrMnNT1lxLpxTp55zsg9nypQprZLsT6Va/orSztW1Ql3ZaUyJ8mE+ZfiIFGaqnTcB4NasPJt2NwmmF5l+NOKsOn9b06mLynQfGkr8AFopvN7/Wh4KzMcAWGs8Yb8j6C4lCORbBb/rb+HraAMHqlLo7HQTdJbXj1i61KmWv6J0AHWt0NEfdYNYwcelERTjJ6jFUe28CW9EQys/15KIAZ04+QH51TYeXvcY35S7eHzILQDkFywhJ3suHm/+BZm6+VSLt+XyzKf7yCt3c9fxt3D4GzpoTRF6/fNALOSnp5BaXgRAMCaIVta0/XoiNp7xkyJDn9clk79xMSjuKobEPcfhqU+fthP7UqKSv6J0IN32e6nqbyWrSwWz+BulommnZt0Ux5oo4t6vFxFhKMV3cCP3706nJmE9PzT9HQuh0kjd1M1w5vnqv4/F23KZ/f53uP2h8o3d33ilMr/LyJaqy/jg4FRKroxB7+Uk99tEunhPUHl9AOe/BAZfw5WObjYw//prQcr6GUD7HSlmWPJXOG6cqxL/SVTyV5QOJDb6DvQ9a6nq4eEm69u8zH/gE5b6920BD7Nz5gFQcCiRkd8+htVbhscSQ7eu3/H7q/P47wiNGKOVyU4/WY4AwaCbNVue5ImNFqaN785nZp1cr59Ui4nZ6cnfe7hnfsESAgW/489jSynxxPD+gSlUHY0gKtAwtvPTr69iubgOvzTR3VDMHmsqLxqvp+eub3H5zQzrdYykHA+BGgOaI0Bwko8b+39A5oLLAZDSz3fxH+OY0bZLSl6MVPJXlA7EfufTTH4ylQ+/zuRV/00M67Kbzb3T8WpRAFgDoTp5yaFISjdbsAVDa//avGUkH3Tx7wYneRmjcNvtBN01fBrcQ1S3eEb0W0fZGCNLjv8WR81+EswBrNVWVi67gfzSUUTGWhg5rQe9RiTVl3G6WNdwU+/lOC1l2E4qHy3PWc7/bnqKQk85yZ4YKoyx1Gge7JHL0Yamk7nJiEnq7HP0ZBXj8EsjPzSup0fSYT7aX0nc9mxcuoU+UYWMTD6GaUDDGKeAQVC+/YrQ82AFO+M/5vCQwyrxhyFa406y7yMrK0vWzXGhKMo52LEAlv4Xo6uexp0SQ17/JDA0tP6tAQ8LHn6ASFd5k49W220su/76+tearlOqb6EwtjN55VfQKXkn0zOWE2vy4vFLehyqpvcJD3nmRH7f/ad8GTuRUXuqMLCK6f3eqe+sBfDrRhbk/IhN+lGI3ESMO4ZKS2WjOYm0oEbysctJPBbDVzEj0A1GfmPYQGenpHdNFp8dfJNqPVSSurvHNzjNjRedByjVElns+xXP9fwrVs3KnNPM19MRCSG2SCnD30Bw8n4q+StKB7RjAYuXfcjPB81ChlnKccXPf9RkqN8Xw0bxz2kzKYyNI06WMiZvI0nZfjTdxM7II0wo9BBlHMJloh8R0orPUEFJ/3/hjv+axCIPxXFWzKWjiDt4MxZPDLq1hKKeC6lK2VB/DJfbzkP6i1RERuOsKue2r+bzG/t8CjSNF2KcfBQZgVW3U3zscab6vsQSzOKe4FYsxvE4+IrXsr+p/65f9lkbdlUtiWBgty6kRIRW+LqUEj+o5K8oCpC0chunZkhDXjXvzP01ndxl9du+GDaKuT++B6+l4QrBKL1Myf2CpIN+HBU9SLZXMTrYDxMNc+J4CfIUHjYZyrg3ehuTyydiCDZ8hxRe8vu/Wn8CkBIeOGYnYIyjIu7nBCy9QjEF3USXvYOh5kuCWhwPlxj5SdkmNlc9gdMwCLtmIEr7B28fClATCHUKN9fyL5AxXGOYx/b/ufo8/ILtz9kmf1XzV5QOrJNWw4lg4xW5jAeqeK3fNczavhBr7eyXL02b2SjxA+jCwqqk4Xxz8N94xXkT8f6hLDJvwCU8REgrWXo6PYPJPBi08UW5kfKyicyLMbDsSgcnrKJ2sRIjU3ZOJ/rYR7yZdjV/j7uL4rREhlT+nYqS31MRCBKlGbA5x7Iz5t8w5l3HjUVrkaWXsUR7hCh9H1sqX6Ba9xJp9HJZQjHzHdeyKdiHdBHgBrm+0bnNHzTwB/1HzLmp/wX/bds7lfwVpQN7rHdffrEzB7/W8E9deAKs7hKaO+bO3R+T4C6nMDb8AiQVWhxOUUkFkWwyZRMQQRIScujWfTu6pZp9nmiS9t+MqBxGlBTYSiUDvz3I9QeXEOWq4LjDyftd+pFW/CZjS6O41gBuQzH/l+xnhVMCgsqAxF22kgEAtplkFI/HaTISG9zH1rKPCcggIKjSrWw6kcy/JX0A9ht5JPBThIBp2nokUBZwsLGgG6v7RPDi4NQL/dO2e+oOX0XpwGYkxTJ1SzXO6gBISURVBY7adWRXdxnKnZN+y+Qb5iI9TZdqBDAEQou4C2R94s/otQGrtTrU4raVc6L/a0R2CZV1TAiGHzLhdFUggJQYJ53LhzPQGIVdEwghsEsbs/JuZ2xFQ2XCLwXuitWM2+VBQ6OfVWNX2ZraxN9AlxrfFHXmIeMC3Fh4Rr8FIUKJ//X9Q9kfuJWnpg86/z9kB6Ra/orSwf1ibG8GvLYTKQ24y/6XktirWBjVj5PXITfvL8M3MBoMJ3UOB72klswHQp2oEFrS8dQ1faXRR2LmB1QdC42tF4YoXut8GyPLNjA0MJq+ZgvGU/odTBh5oPR6pvVfQ1lAsKzCSNl+J9bCf+IOVvEvnxNHoDLcOidU6RZSROiklEccALFaNZrjGiwJvZmcfualKpUWtvyFEDcJIXYJIYJCiGY7GIQQ1wgh9gkhDgohHm7JMRVFOTe9RiTxgzsHEBFrodroIL70S35YuZtEKRESEqVkXMEXOEteCi1iLiUGvZjYkpeYc+wjAJyEOlktluqwxzDaS+ufVwpJlSmSlfFj8XtisTWTZUyeWISAWKPkmhIbY3bFQbAKAURUVyCaWeok0uglT4aSfgqhk0BVIAHN2pvxMwZ8n5/oktTSlv9OYDrwj+Z2EEJowF+BicBxYJMQ4kMp5e4WHltRlLPUa0QSvUYk8eZbJ8hd9gbxpV9yS+mXAGgiiCMRKrcYKUh7CJfDT5w/hnuLM7jGEwABE1jHUibidDEPcQAACg5JREFU9TqwWpueAPSa0F2+fiRrrKGx/brBRJUhiDsI9jCLZmmiuP550aYEDIFTzxKC0FInDe1/owgwPOE4f9JvxoaXXxvfxR+08LXrVqoi99FrhGr1n60WJX8p5R4446ozw4GDUsqc2n3nA9MAlfwVpZX95Lbp3LAll8En1uII1FBjsBNTrfPjwBrcFZPZuS2On27fQISrEKM9H9fIIOZUAwN9+6gRJjYfyqRXr28alX6CuonCHTdSIYKsserstTS896VFZ4BHMtBubVT6kdJPlPH1+td+V/MLv1ttPjxuE5FGLwPiS3jVfiOrvJn8wfpPxrOLVZU/54DnSqxdd5znX6tja42afypw7KTXx4ER4XYUQtwD3AOQlpZ24SNTlEvQnbdPZ/b7GUwMfMlDxgWkiBLyZSyp+g5+YjnE01f9J25DBL8++jJRXj8FTjO7ejsRhgL6UrsATG1FxuC2cGJjJM/UdKfK2XTM/V5zgHjjG+x1zyTdEoPNAJJK4kzzMDjWAqErBlOEH7/L3OTzpgg/fX6cXX/MlE//yRUFH5NWs4Qjlgm8ZrwVzZRK0OBRUzWfozMmfyHEF0BSmLd+I6VcchbHCHdZELaYJ6WcB8yD0E1eZ/HdiqKcoxtqh0H+4Y0yxphHk+Au445dHzM+dxvv9UthVcon1Jg8xAaGMLYymZG8TX8qyOnuwGMxYPEGsezpyqDK3ZiEDzTYF/0ub9TcgZQNCVwIP1bgX6ZuPG2/G7toWETFjYnsbvb610lDSzi6LhlxUl+yMAZJHl5U/1pzx7Cp+GOOVh8AQzzCYEMzpaIbfPQa6FAzdp6jMyZ/KWVLi2jHgS4nve4MNF35WFGUVnPD4FTGHY8g/9FHkZ6GRVFG7DMwak81mpTAt+RGZ7OifxojOcTIojIKNI1/xjoRFaU4Cw10jzRgtAd5tHo1l+HlSe0OygKRWISf+zYvRAsGeXHIDwEarjKI5ZtOKUQm5IMEb7WZL6PHMuSqTVR/Y8XnMmN2+EkeUUhsRiUAQb/g8HorLs9BMMRjoArN0ocqcympgw1ce9c1bfEztmutUfbZBGQIIboDucBM4NZWOK6iKKdRtwRh4XPP48vLpzA2jpemhRZ0+eU7L2Pzekgpd3Fkr5NXXaMIEqTaGmBL7zIO9ajh3S69mHDoCh41vEm8tYprLRtI6naI1dYruat8DWuM3Uj+opL/3LqQVwdO5grraCKFjxEeA0ZLPJ8flXjcAQLdIyBJ48ZOixEZ8NCaOYy2fEXPpCNICbpLw7bPxLeM4au0ftx++B2GpWbzUN9f8TP7EH5y6ytt+TO2Wy2a20cIcSPwZyABKAe2SyknCSFSgJeklNfV7ncd8DygAa9IKX9/pu9Wc/soSuvJWr+L415/k+1OX5CgbwOWspcRsqFsI4MmPPnT0SsHExUQ3BKw8+ep0QB0tpjYPKp/oxXBjJqTl45NYd3B4fSu0LjGbcJ08igePPSe+gABm87XeUN5ddePCcjGbVMNnbsGvM2Nm7eRHlVG6YD7Sb15zoX5QdoxNbGboihnbVFBKQ/uO4Y72JAPrAGJVTNQjsTi+gpH6QIMshTpj8ZbNAm9cjBGCRM8Jg4OdrKrqwWbQTC3d5dmF3hZVFDKUzn5RO93MXlHGeYaE0Z7KRGR67mi07vkZFgJaoKv84Yyf98MXP7QvEQOYzU/6rOIsXFbuCL1KTU//2mo5K8oyjmpS8wnr9J1/56jjUZnGPKqMR6oQngCOANwpcHKiYFO1qUaW7Sy1+uPfIVuW8nIxNco7RLAa6m9MeCk4SIGg40+fX5/QdcT7ghU8lcUpcWaKwd1Kili/qOz6Lvn/Nyus39jAave3ovua5jLJ7r7RlKHLyUgC1tlIfmOQk3prChKi81OT+aX3x7Ea24Ywmnxerl7yXyMycnn7Ti9RoRGk3+9JBtXqZeIWAvDxt5FrxGzz9sxlMZU8lcUpVkzkmJxb6vimVJJYUwsiaUl3L1kPhO/20rik0+c12PVTUGhtA6V/BVFOa3brp3A1KVLKfzdr9Hz8zEmJ5P45BP1Q0WV9kklf0VRzsg5dapK9h2MWsxFURTlEqSSv6IoyiVIJX9FUZRLkEr+iqIolyCV/BVFUS5BKvkriqJcglTyVxRFuQSp5K8oinIJumgndhNCFAFHzsNXxQPF5+F72oqKv+2059ihfcffnmOHto2/q5Qy4Uw7XbTJ/3wRQmw+mxnuLlYq/rbTnmOH9h1/e44d2kf8quyjKIpyCVLJX1EU5RJ0KST/eW0dQAup+NtOe44d2nf87Tl2aAfxd/iav6IoitLUpdDyVxRFUU7RoZO/EOIaIcQ+IcRBIcTDbR3PuRBCvCKEKBRC7GzrWM6VEKKLEGKVEGKPEGKXEGJWW8d0LoQQViHEN0KIb2vjf7ytYzpXQghNCLFNCLGsrWM5V0KIw0KI74QQ24UQ7WohbyFEtBBioRBib+3//yPbOqbmdNiyjxBCA/YDE4HjwCbgR1LK87Pi9AUmhLgScAFvSCkHtHU850IIkQwkSym3CiEigS3ADe3otxeAQ0rpEkKYgHXALCnlhjYO7awJIX4JZAFRUsopbR3PuRBCHAaypJTtbpy/EOJ1YK2U8iUhhBmwSynL2zqucDpyy384cFBKmSOl9AHzgWltHNNZk1KuAUrbOo7vQ0qZL6XcWvu8CtgDpLZtVGdPhrhqX5pq/2s3rSQhRGdgMvBSW8dyKRFCRAFXAi8DSCl9F2vih46d/FOBYye9Pk47SkAdhRCiGzAY2Ni2kZyb2rLJdqAQ+FxK2Z7ifx54CAi2dSDfkwQ+E0JsEULc09bBnIN0oAh4tbbk9pIQwtHWQTWnIyd/EWZbu2m9dQRCiAhgEfALKWVlW8dzLqSUASnlIKAzMFwI0S5Kb0KIKUChlHJLW8fSAqOllEOAa4H7akug7YERGAL8TUo5GKgGLtq+xo6c/I8DXU563RnIa6NYLjm1tfJFwNtSyvfbOp7vq/ayfTVwTRuHcrZGA9fX1s3nA+OFEG+1bUjnRkqZV/tYCHxAqITbHhwHjp90lbiQ0MngotSRk/8mIEMI0b2242Um8GEbx3RJqO0wfRnYI6X837aO51wJIRKEENG1z23AD4C9bRvV2ZFSzpZSdpZSdiP0//xKKeVtbRzWWRNCOGoHCVBbMrkaaBcj3qSUBcAxIUTv2k0TgIt2kIOxrQO4UKSUuhDifuBTQANekVLuauOwzpoQ4l/AWCBeCHEc+B8p5cttG9VZGw38BPiutm4O8IiU8qM2jOlcJAOv144YMwALpJTtbshkO9UJ+CDUfsAIvCOl/KRtQzon/wm8XdvgzAHuauN4mtVhh3oqiqIozevIZR9FURSlGSr5K4qiXIJU8lcURbkEqeSvKIpyCVLJX1EU5RKkkr+iKMolSCV/RVGUS5BK/oqiKJeg/wdx8IWXHsvymwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# declare the model\n",
    "model = SineModel().to(device)\n",
    "\n",
    "# define the criterion\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "# select the optimizer and pass to it the parameters of the model it will optimize\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr = 0.01)\n",
    "\n",
    "epochs = 1000\n",
    "\n",
    "# training loop\n",
    "for epoch in range(epochs):\n",
    "    for i, (x_i, y_i) in enumerate(sine_loader):\n",
    "\n",
    "        y_hat_i = model(x_i)            # forward pass\n",
    "                                \n",
    "        loss = criterion(y_hat_i, y_i)  # compute the loss and perform the backward pass\n",
    "\n",
    "        optimizer.zero_grad()           # cleans the gradients\n",
    "        loss.backward()                 # computes the gradients\n",
    "        optimizer.step()                # update the parameters\n",
    "\n",
    "    if epoch % 20:\n",
    "        plt.scatter(x_i.data.cpu().numpy(), y_hat_i.data.cpu().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.00704681989736855\n"
     ]
    }
   ],
   "source": [
    "# testing\n",
    "with torch.no_grad():\n",
    "    model.eval()\n",
    "    total_loss = 0.\n",
    "    for k, (x_k, y_k) in enumerate(sine_loader_test):\n",
    "        y_hat_k = model(x_k)\n",
    "        loss_test = criterion(y_hat_k, y_k)\n",
    "        total_loss += float(loss_test)\n",
    "\n",
    "print(total_loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reproducibility <a id=\"reproducibility\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def enforce_reproducibility(seed=42):\n",
    "    # Sets seed manually for both CPU and CUDA\n",
    "    torch.manual_seed(seed)\n",
    "    # For atomic operations there is currently \n",
    "    # no simple way to enforce determinism, as\n",
    "    # the order of parallel operations is not known.\n",
    "    #\n",
    "    # CUDNN\n",
    "    torch.backends.cudnn.deterministic = True\n",
    "    torch.backends.cudnn.benchmark = False\n",
    "    # System based\n",
    "    np.random.seed(seed)\n",
    "\n",
    "enforce_reproducibility()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References and Further Reading <a id=\"references\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lapan, Maxim (2018) *Deep Reinforcement Learning Hands-On*. Birmingham: Packt Publishing\n",
    "\n",
    "Rao, Delip and Brian McMahan (2019) *Natural Language Processing with PyTorch*. Sebastopol, CA: O'Reilly Media"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}