slides18/neural_networks_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.Tensor(torch.randn(2,2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       " 2.3847  0.3085\n",
       "-2.3289  0.4508\n",
       "[torch.FloatTensor of size 2x2]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Neural Networks\n",
    "===============\n",
    "\n",
    "Neural networks can be constructed using the ``torch.nn`` package.\n",
    "\n",
    "Now that you had a glimpse of ``autograd``, ``nn`` depends on\n",
    "``autograd`` to define models and differentiate them.\n",
    "An ``nn.Module`` contains layers, and a method ``forward(input)``\\ that\n",
    "returns the ``output``.\n",
    "\n",
    "For example, look at this network that classifies digit images:\n",
    "\n",
    ".. figure:: /_static/img/mnist.png\n",
    "   :alt: convnet\n",
    "\n",
    "   convnet\n",
    "\n",
    "It is a simple feed-forward network. It takes the input, feeds it\n",
    "through several layers one after the other, and then finally gives the\n",
    "output.\n",
    "\n",
    "A typical training procedure for a neural network is as follows:\n",
    "\n",
    "- Define the neural network that has some learnable parameters (or\n",
    "  weights)\n",
    "- Iterate over a dataset of inputs\n",
    "- Process input through the network\n",
    "- Compute the loss (how far is the output from being correct)\n",
    "- Propagate gradients back into the network’s parameters\n",
    "- Update the weights of the network, typically using a simple update rule:\n",
    "  ``weight = weight - learning_rate * gradient``\n",
    "\n",
    "Define the network\n",
    "------------------\n",
    "\n",
    "Let’s define this network:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "\n",
    "class Net(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        # 1 input image channel, 6 output channels, 5x5 square convolution\n",
    "        # kernel\n",
    "        self.conv1 = nn.Conv2d(1, 6, kernel_size=5)\n",
    "        self.conv2 = nn.Conv2d(6, 16, kernel_size=5)\n",
    "        # an affine operation: y = Wx + b\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, out_features=120)\n",
    "        self.fc2 = nn.Linear(120, out_features=84)\n",
    "        self.fc3 = nn.Linear(84, out_features=10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Max pooling over a (2, 2) window\n",
    "        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))\n",
    "        # If the size is a square you can only specify a single number\n",
    "        x = F.max_pool2d(F.relu(self.conv2(x)), 2)\n",
    "        x = x.view(-1, self.num_flat_features(x))\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x\n",
    "\n",
    "    def num_flat_features(self, x):\n",
    "        size = x.size()[1:]  # all dimensions except the batch dimension\n",
    "        num_features = 1\n",
    "        for s in size:\n",
    "            num_features *= s\n",
    "        return num_features\n",
    "\n",
    "\n",
    "net = Net()\n",
    "net = net.cuda()\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for m in net.modules():\n",
    "    print(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "net."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You just have to define the ``forward`` function, and the ``backward``\n",
    "function (where gradients are computed) is automatically defined for you\n",
    "using ``autograd``.\n",
    "You can use any of the Tensor operations in the ``forward`` function.\n",
    "\n",
    "The learnable parameters of a model are returned by ``net.parameters()``\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([6, 1, 5, 5])\n",
      "torch.Size([6])\n",
      "torch.Size([16, 6, 5, 5])\n",
      "torch.Size([16])\n",
      "torch.Size([120, 400])\n",
      "torch.Size([120])\n",
      "torch.Size([84, 120])\n",
      "torch.Size([84])\n",
      "torch.Size([10, 84])\n",
      "torch.Size([10])\n",
      "10\n",
      "torch.Size([6, 1, 5, 5])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['__add__',\n",
       " '__and__',\n",
       " '__bool__',\n",
       " '__class__',\n",
       " '__deepcopy__',\n",
       " '__delattr__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__div__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__float__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattr__',\n",
       " '__getattribute__',\n",
       " '__getitem__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__iadd__',\n",
       " '__iand__',\n",
       " '__idiv__',\n",
       " '__ilshift__',\n",
       " '__imul__',\n",
       " '__init__',\n",
       " '__int__',\n",
       " '__ior__',\n",
       " '__ipow__',\n",
       " '__irshift__',\n",
       " '__isub__',\n",
       " '__iter__',\n",
       " '__ixor__',\n",
       " '__le__',\n",
       " '__len__',\n",
       " '__long__',\n",
       " '__lshift__',\n",
       " '__lt__',\n",
       " '__matmul__',\n",
       " '__mod__',\n",
       " '__module__',\n",
       " '__mul__',\n",
       " '__ne__',\n",
       " '__neg__',\n",
       " '__new__',\n",
       " '__nonzero__',\n",
       " '__or__',\n",
       " '__pow__',\n",
       " '__radd__',\n",
       " '__rdiv__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__rmul__',\n",
       " '__rpow__',\n",
       " '__rshift__',\n",
       " '__rsub__',\n",
       " '__rtruediv__',\n",
       " '__setattr__',\n",
       " '__setitem__',\n",
       " '__setstate__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__sub__',\n",
       " '__subclasshook__',\n",
       " '__truediv__',\n",
       " '__weakref__',\n",
       " '__xor__',\n",
       " '_advanced_index_add',\n",
       " '_backward_hooks',\n",
       " '_execution_engine',\n",
       " '_fallthrough_methods',\n",
       " '_get_type',\n",
       " '_grad',\n",
       " '_grad_fn',\n",
       " '_torch',\n",
       " '_unnarrow',\n",
       " '_version',\n",
       " 'abs',\n",
       " 'abs_',\n",
       " 'acos',\n",
       " 'acos_',\n",
       " 'add',\n",
       " 'add_',\n",
       " 'addbmm',\n",
       " 'addbmm_',\n",
       " 'addcdiv',\n",
       " 'addcdiv_',\n",
       " 'addcmul',\n",
       " 'addcmul_',\n",
       " 'addmm',\n",
       " 'addmm_',\n",
       " 'addmv',\n",
       " 'addmv_',\n",
       " 'addr',\n",
       " 'addr_',\n",
       " 'all',\n",
       " 'any',\n",
       " 'asin',\n",
       " 'asin_',\n",
       " 'atan',\n",
       " 'atan2',\n",
       " 'atan2_',\n",
       " 'atan_',\n",
       " 'backward',\n",
       " 'baddbmm',\n",
       " 'baddbmm_',\n",
       " 'bernoulli',\n",
       " 'bmm',\n",
       " 'btrifact',\n",
       " 'btrisolve',\n",
       " 'byte',\n",
       " 'cat',\n",
       " 'ceil',\n",
       " 'ceil_',\n",
       " 'char',\n",
       " 'chunk',\n",
       " 'clamp',\n",
       " 'clone',\n",
       " 'contiguous',\n",
       " 'cos',\n",
       " 'cos_',\n",
       " 'cosh',\n",
       " 'cosh_',\n",
       " 'cpu',\n",
       " 'cross',\n",
       " 'cuda',\n",
       " 'cumprod',\n",
       " 'cumsum',\n",
       " 'data',\n",
       " 'data_ptr',\n",
       " 'detach',\n",
       " 'detach_',\n",
       " 'diag',\n",
       " 'dist',\n",
       " 'div',\n",
       " 'div_',\n",
       " 'dot',\n",
       " 'double',\n",
       " 'eig',\n",
       " 'eq',\n",
       " 'eq_',\n",
       " 'equal',\n",
       " 'erf',\n",
       " 'erf_',\n",
       " 'erfinv',\n",
       " 'erfinv_',\n",
       " 'exp',\n",
       " 'exp_',\n",
       " 'expand',\n",
       " 'expand_as',\n",
       " 'fill_',\n",
       " 'float',\n",
       " 'floor',\n",
       " 'floor_',\n",
       " 'fmod',\n",
       " 'fmod_',\n",
       " 'frac',\n",
       " 'frac_',\n",
       " 'gather',\n",
       " 'ge',\n",
       " 'ge_',\n",
       " 'gels',\n",
       " 'geometric_',\n",
       " 'geqrf',\n",
       " 'ger',\n",
       " 'gesv',\n",
       " 'get_device',\n",
       " 'grad',\n",
       " 'grad_fn',\n",
       " 'gt',\n",
       " 'gt_',\n",
       " 'half',\n",
       " 'histc',\n",
       " 'index_add',\n",
       " 'index_add_',\n",
       " 'index_copy',\n",
       " 'index_copy_',\n",
       " 'index_fill',\n",
       " 'index_fill_',\n",
       " 'index_select',\n",
       " 'int',\n",
       " 'inverse',\n",
       " 'is_contiguous',\n",
       " 'is_leaf',\n",
       " 'is_same_size',\n",
       " 'is_set_to',\n",
       " 'kthvalue',\n",
       " 'le',\n",
       " 'le_',\n",
       " 'lerp',\n",
       " 'lerp_',\n",
       " 'lgamma',\n",
       " 'lgamma_',\n",
       " 'log',\n",
       " 'log1p',\n",
       " 'log1p_',\n",
       " 'log_',\n",
       " 'log_normal_',\n",
       " 'long',\n",
       " 'lt',\n",
       " 'lt_',\n",
       " 'masked_copy',\n",
       " 'masked_copy_',\n",
       " 'masked_fill',\n",
       " 'masked_fill_',\n",
       " 'masked_scatter',\n",
       " 'masked_scatter_',\n",
       " 'masked_select',\n",
       " 'matmul',\n",
       " 'max',\n",
       " 'mean',\n",
       " 'median',\n",
       " 'min',\n",
       " 'mm',\n",
       " 'mode',\n",
       " 'mul',\n",
       " 'mul_',\n",
       " 'multinomial',\n",
       " 'mv',\n",
       " 'narrow',\n",
       " 'ne',\n",
       " 'ne_',\n",
       " 'neg',\n",
       " 'neg_',\n",
       " 'nonzero',\n",
       " 'norm',\n",
       " 'numel',\n",
       " 'ones_like',\n",
       " 'orgqr',\n",
       " 'ormqr',\n",
       " 'output_nr',\n",
       " 'permute',\n",
       " 'potrf',\n",
       " 'potri',\n",
       " 'potrs',\n",
       " 'pow',\n",
       " 'pow_',\n",
       " 'prod',\n",
       " 'pstrf',\n",
       " 'put_',\n",
       " 'qr',\n",
       " 'reciprocal',\n",
       " 'reciprocal_',\n",
       " 'register_hook',\n",
       " 'reinforce',\n",
       " 'remainder',\n",
       " 'remainder_',\n",
       " 'renorm',\n",
       " 'repeat',\n",
       " 'requires_grad',\n",
       " 'resize',\n",
       " 'resize_as',\n",
       " 'retain_grad',\n",
       " 'round',\n",
       " 'round_',\n",
       " 'rsqrt',\n",
       " 'rsqrt_',\n",
       " 'scatter',\n",
       " 'scatter_',\n",
       " 'scatter_add',\n",
       " 'scatter_add_',\n",
       " 'select',\n",
       " 'set_',\n",
       " 'short',\n",
       " 'sigmoid',\n",
       " 'sigmoid_',\n",
       " 'sign',\n",
       " 'sign_',\n",
       " 'sin',\n",
       " 'sin_',\n",
       " 'sinh',\n",
       " 'sinh_',\n",
       " 'sort',\n",
       " 'split',\n",
       " 'sqrt',\n",
       " 'sqrt_',\n",
       " 'squeeze',\n",
       " 'squeeze_',\n",
       " 'std',\n",
       " 'storage_offset',\n",
       " 'sub',\n",
       " 'sub_',\n",
       " 'sum',\n",
       " 'svd',\n",
       " 'symeig',\n",
       " 't',\n",
       " 't_',\n",
       " 'take',\n",
       " 'tan',\n",
       " 'tan_',\n",
       " 'tanh',\n",
       " 'tanh_',\n",
       " 'topk',\n",
       " 'trace',\n",
       " 'transpose',\n",
       " 'transpose_',\n",
       " 'tril',\n",
       " 'tril_',\n",
       " 'triu',\n",
       " 'triu_',\n",
       " 'trtrs',\n",
       " 'trunc',\n",
       " 'trunc_',\n",
       " 'type',\n",
       " 'type_as',\n",
       " 'unfold',\n",
       " 'unsqueeze',\n",
       " 'unsqueeze_',\n",
       " 'var',\n",
       " 'view',\n",
       " 'view_as',\n",
       " 'volatile',\n",
       " 'zero_',\n",
       " 'zeros_like']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = list(net.parameters())\n",
    "for p in params:\n",
    "    print(p.shape)\n",
    "print(len(params))\n",
    "print(params[0].size())  # conv1's .weight\n",
    "\n",
    "dir(params[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The input to the forward is an ``autograd.Variable``, and so is the output.\n",
    "Note: Expected input size to this net(LeNet) is 32x32. To use this net on\n",
    "MNIST dataset, please resize the images from the dataset to 32x32.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params[0].get_device()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 0.0172  0.1396 -0.0320 -0.0748 -0.0438  0.1131 -0.0328 -0.1025 -0.0912 -0.0820\n",
      "[torch.cuda.FloatTensor of size 1x10 (GPU 0)]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "input = Variable(torch.randn(1, 1, 32, 32)).cuda()\n",
    "out = net(input)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zero the gradient buffers of all parameters and backprops with random\n",
    "gradients:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "Expected object of type Variable[torch.FloatTensor] but found type Variable[torch.cuda.FloatTensor] for argument #1 'mat2'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-7ef080f1cd99>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mnet\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrandn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/torch/autograd/variable.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph, retain_variables)\u001b[0m\n\u001b[1;32m    165\u001b[0m                 \u001b[0mVariable\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    166\u001b[0m         \"\"\"\n\u001b[0;32m--> 167\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_variables\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    168\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    169\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(variables, grad_variables, retain_graph, create_graph, retain_variables)\u001b[0m\n\u001b[1;32m     97\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     98\u001b[0m     Variable._execution_engine.run_backward(\n\u001b[0;32m---> 99\u001b[0;31m         variables, grad_variables, retain_graph)\n\u001b[0m\u001b[1;32m    100\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: Expected object of type Variable[torch.FloatTensor] but found type Variable[torch.cuda.FloatTensor] for argument #1 'mat2'"
     ]
    }
   ],
   "source": [
    "net.zero_grad()\n",
    "out.backward(torch.randn(1, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\"><h4>Note</h4><p>``torch.nn`` only supports mini-batches. The entire ``torch.nn``\n",
    "    package only supports inputs that are a mini-batch of samples, and not\n",
    "    a single sample.\n",
    "\n",
    "    For example, ``nn.Conv2d`` will take in a 4D Tensor of\n",
    "    ``nSamples x nChannels x Height x Width``.\n",
    "\n",
    "    If you have a single sample, just use ``input.unsqueeze(0)`` to add\n",
    "    a fake batch dimension.</p></div>\n",
    "\n",
    "Before proceeding further, let's recap all the classes you’ve seen so far.\n",
    "\n",
    "**Recap:**\n",
    "  -  ``torch.Tensor`` - A *multi-dimensional array*.\n",
    "  -  ``autograd.Variable`` - *Wraps a Tensor and records the history of\n",
    "     operations* applied to it. Has the same API as a ``Tensor``, with\n",
    "     some additions like ``backward()``. Also *holds the gradient*\n",
    "     w.r.t. the tensor.\n",
    "  -  ``nn.Module`` - Neural network module. *Convenient way of\n",
    "     encapsulating parameters*, with helpers for moving them to GPU,\n",
    "     exporting, loading, etc.\n",
    "  -  ``nn.Parameter`` - A kind of Variable, that is *automatically\n",
    "     registered as a parameter when assigned as an attribute to a*\n",
    "     ``Module``.\n",
    "  -  ``autograd.Function`` - Implements *forward and backward definitions\n",
    "     of an autograd operation*. Every ``Variable`` operation, creates at\n",
    "     least a single ``Function`` node, that connects to functions that\n",
    "     created a ``Variable`` and *encodes its history*.\n",
    "\n",
    "**At this point, we covered:**\n",
    "  -  Defining a neural network\n",
    "  -  Processing inputs and calling backward\n",
    "\n",
    "**Still Left:**\n",
    "  -  Computing the loss\n",
    "  -  Updating the weights of the network\n",
    "\n",
    "Loss Function\n",
    "-------------\n",
    "A loss function takes the (output, target) pair of inputs, and computes a\n",
    "value that estimates how far away the output is from the target.\n",
    "\n",
    "There are several different\n",
    "`loss functions <http://pytorch.org/docs/nn.html#loss-functions>`_ under the\n",
    "nn package .\n",
    "A simple loss is: ``nn.MSELoss`` which computes the mean-squared error\n",
    "between the input and the target.\n",
    "\n",
    "For example:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "output = net(input)\n",
    "target = Variable(torch.arange(1, 11))  # a dummy target, for example\n",
    "target = target.view(1, -1)  # make it the same shape as output\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "loss = criterion(output, target)\n",
    "print(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, if you follow ``loss`` in the backward direction, using its\n",
    "``.grad_fn`` attribute, you will see a graph of computations that looks\n",
    "like this:\n",
    "\n",
    "::\n",
    "\n",
    "    input -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d\n",
    "          -> view -> linear -> relu -> linear -> relu -> linear\n",
    "          -> MSELoss\n",
    "          -> loss\n",
    "\n",
    "So, when we call ``loss.backward()``, the whole graph is differentiated\n",
    "w.r.t. the loss, and all Variables in the graph will have their\n",
    "``.grad`` Variable accumulated with the gradient.\n",
    "\n",
    "For illustration, let us follow a few steps backward:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(loss.grad_fn)  # MSELoss\n",
    "print(loss.grad_fn.next_functions[0][0])  # Linear\n",
    "print(loss.grad_fn.next_functions[0][0].next_functions[0][0])  # ReLU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Backprop\n",
    "--------\n",
    "To backpropagate the error all we have to do is to ``loss.backward()``.\n",
    "You need to clear the existing gradients though, else gradients will be\n",
    "accumulated to existing gradients.\n",
    "\n",
    "\n",
    "Now we shall call ``loss.backward()``, and have a look at conv1's bias\n",
    "gradients before and after the backward.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "net.zero_grad()     # zeroes the gradient buffers of all parameters\n",
    "\n",
    "print('conv1.bias.grad before backward')\n",
    "print(net.conv1.bias.grad)\n",
    "\n",
    "loss.backward()\n",
    "\n",
    "print('conv1.bias.grad after backward')\n",
    "print(net.conv1.bias.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we have seen how to use loss functions.\n",
    "\n",
    "**Read Later:**\n",
    "\n",
    "  The neural network package contains various modules and loss functions\n",
    "  that form the building blocks of deep neural networks. A full list with\n",
    "  documentation is `here <http://pytorch.org/docs/nn>`_.\n",
    "\n",
    "**The only thing left to learn is:**\n",
    "\n",
    "  - Updating the weights of the network\n",
    "\n",
    "Update the weights\n",
    "------------------\n",
    "The simplest update rule used in practice is the Stochastic Gradient\n",
    "Descent (SGD):\n",
    "\n",
    "     ``weight = weight - learning_rate * gradient``\n",
    "\n",
    "We can implement this using simple python code:\n",
    "\n",
    ".. code:: python\n",
    "\n",
    "    learning_rate = 0.01\n",
    "    for f in net.parameters():\n",
    "        f.data.sub_(f.grad.data * learning_rate)\n",
    "\n",
    "However, as you use neural networks, you want to use various different\n",
    "update rules such as SGD, Nesterov-SGD, Adam, RMSProp, etc.\n",
    "To enable this, we built a small package: ``torch.optim`` that\n",
    "implements all these methods. Using it is very simple:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "# create your optimizer\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01)\n",
    "\n",
    "# in your training loop:\n",
    "optimizer.zero_grad()   # zero the gradient buffers\n",
    "output = net(input)\n",
    "loss = criterion(output, target)\n",
    "loss.backward()\n",
    "optimizer.step()    # Does the update"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ".. Note::\n",
    "\n",
    "      Observe how gradient buffers had to be manually set to zero using\n",
    "      ``optimizer.zero_grad()``. This is because gradients are accumulated\n",
    "      as explained in `Backprop`_ section.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "5cpu-1gpu-50G-python",
   "language": "python",
   "name": "5cpu-1gpu-50g-python"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}