auto_grad.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Autograd: automatic differentiation\n",
    "===================================\n",
    "\n",
    "Central to all neural networks in PyTorch is the ``autograd`` package.\n",
    "Let’s first briefly visit this, and we will then go to training our\n",
    "first neural network.\n",
    "\n",
    "\n",
    "The ``autograd`` package provides automatic differentiation for all operations\n",
    "on Tensors. It is a define-by-run framework, which means that your backprop is\n",
    "defined by how your code is run, and that every single iteration can be\n",
    "different.\n",
    "\n",
    "Let us see this in more simple terms with some examples.\n",
    "\n",
    "Variable\n",
    "--------\n",
    "\n",
    "``autograd.Variable`` is the central class of the package. It wraps a\n",
    "Tensor, and supports nearly all of operations defined on it. Once you\n",
    "finish your computation you can call ``.backward()`` and have all the\n",
    "gradients computed automatically.\n",
    "\n",
    "You can access the raw tensor through the ``.data`` attribute, while the\n",
    "gradient w.r.t. this variable is accumulated into ``.grad``.\n",
    "\n",
    ".. figure:: /_static/img/Variable.png\n",
    "   :alt: Variable\n",
    "\n",
    "   Variable\n",
    "\n",
    "There’s one more class which is very important for autograd\n",
    "implementation - a ``Function``.\n",
    "\n",
    "``Variable`` and ``Function`` are interconnected and build up an acyclic\n",
    "graph, that encodes a complete history of computation. Each variable has\n",
    "a ``.creator`` attribute that references a ``Function`` that has created\n",
    "the ``Variable`` (except for Variables created by the user - their\n",
    "``creator is None``).\n",
    "\n",
    "If you want to compute the derivatives, you can call ``.backward()`` on\n",
    "a ``Variable``. If ``Variable`` is a scalar (i.e. it holds a one element\n",
    "data), you don’t need to specify any arguments to ``backward()``,\n",
    "however if it has more elements, you need to specify a ``grad_output``\n",
    "argument that is a tensor of matching shape.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a variable:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 1  1\n",
      " 1  1\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "x = Variable(torch.ones(2, 2), requires_grad=True)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do an operation of variable:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 3  3\n",
      " 3  3\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y = x + 2\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``y`` was created as a result of an operation, so it has a creator.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do more operations on y\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(Variable containing:\n",
      " 27  27\n",
      " 27  27\n",
      "[torch.FloatTensor of size 2x2]\n",
      ", Variable containing:\n",
      " 27\n",
      "[torch.FloatTensor of size 1]\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "\n",
    "print(z, out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradients\n",
    "---------\n",
    "let's backprop now\n",
    "``out.backward()`` is equivalent to doing ``out.backward(torch.Tensor([1.0]))``\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print gradients d(out)/dx\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      " 4.5000  4.5000\n",
      " 4.5000  4.5000\n",
      "[torch.FloatTensor of size 2x2]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should have got a matrix of ``4.5``. Let’s call the ``out``\n",
    "*Variable* “$o$”.\n",
    "We have that $o = \\frac{1}{4}\\sum_i z_i$,\n",
    "$z_i = 3(x_i+2)^2$ and $z_i\\bigr\\rvert_{x_i=1} = 18$.\n",
    "Therefore,\n",
    "$\\frac{\\partial o}{\\partial x_i} = \\frac{3}{2}(x_i+2)$, hence\n",
    "$\\frac{\\partial o}{\\partial x_i}\\bigr\\rvert_{x_i=1} = \\frac{9}{2} = 4.5$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do many crazy things with autograd!\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      "-1020.3505\n",
      " 1633.7242\n",
      " -333.5652\n",
      "[torch.FloatTensor of size 3]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "x = torch.randn(3)\n",
    "x = Variable(x, requires_grad=True)\n",
    "\n",
    "y = x * 2\n",
    "while y.data.norm() < 1000:\n",
    "    y = y * 2\n",
    "\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variable containing:\n",
      "  102.4000\n",
      " 1024.0000\n",
      "    0.1024\n",
      "[torch.FloatTensor of size 3]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "gradients = torch.FloatTensor([0.1, 1.0, 0.0001])\n",
    "y.backward(gradients)\n",
    "\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Read Later:**\n",
    "\n",
    "Documentation of ``Variable`` and ``Function`` is at\n",
    "http://pytorch.org/docs/autograd\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}