lab2/Lab 2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "This project was developed by Peter Chen, Rocky Duan, Pieter Abbeel for the Berkeley Deep RL Bootcamp, August 2017.\n",
    "Bootcamp website with slides and lecture videos: https://sites.google.com/view/deep-rl-bootcamp/.\n",
    "\n",
    "Code adapted from Stanford CS231N materials: http://cs231n.stanford.edu/\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "# note to properly run this lab, you should execute all code blocks sequentially\n",
    "import chainer\n",
    "import chainer.functions as F\n",
    "import chainer.links as L\n",
    "from chainer import training\n",
    "from chainer.training import extensions\n",
    "\n",
    "from collections import namedtuple, defaultdict, deque\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Chainer\n",
    "\n",
    "Chainer can be understood as Numpy plus the ability to record the computation graph of numerical operations to enable Automatic Differentiation. (Chainer actually also offers many other things; for example, a Numpy equivalent library that runs on GPU, but we will ignore them for now)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's illustrate how Chainer works by a simple 1D regression task.\n",
    "\n",
    "Suppose we have observations from the following model $y = w x + b + \\epsilon$ where $\\epsilon \\sim \\mathcal{N}(0, 0.1)$ and the task is to estimate the linear model parameters $w, b$ from data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAD8CAYAAACfF6SlAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAG2xJREFUeJzt3X2sHOd13/HvjxSY6iJ+CV+qCKJ4r9LQQZW0UMwLNUHQ2IXllvYfVIEYjlRKlgq1hOOqCOAmgAIBQaCArV2jeSkiFGGTIjbJVFYEpCFgBWqiWDAQWIauEVuIJMiiVZOiotQ3pGugkBNV1ukfsxsOlzO7szvPzM7s/D7A4t7dO3deKOjMc89znjOKCMzMbFh2LPsEzMysfQ7+ZmYD5OBvZjZADv5mZgPk4G9mNkAO/mZmA+Tgb2Y2QA7+ZmYD5OBvZjZA1yz7BMrs3bs3NjY2ln0aZma98uUvf/mvImLfrO06G/w3NjbY2tpa9mmYmfWKpHNVtnPax8xsgBz8zcwGyMHfzGyAHPzNzAbIwd/MbIAc/M3MBsjB38xsgBz8zcwqOn0aNjZgx47s6+nTyz6jxTn4m5lVcPo0HDsG585BRPb12LH6N4Bl3VAc/M3MKnjwQXj99Ss/e/317PNFNXVDqcLB38ysgvPn5/u8iiZuKFU5+JuZVXDgwHyfV9HEDaUqB38z65SuTqoePw5ra1d+traWfb6oJm4oVTn4m1lnLDMHPsvRo3DiBKyvg5R9PXEi+3xRTdxQqlJENH+UBWxuboZbOpsNy8ZGFvAnra/DN77R9tm04/TpLMd//nw24j9+vN4NRdKXI2Jz1nYe+ZtZZywzB57KvGmro0ezG9tbb2Vf6wT+eTj4m1lnLDMHnkKX01aTkgR/Sf9N0jcl/XnJzyXpP0s6K+lZSe9OcVwzWy3LzIGnsMzSzXmlGvn/DnB4ys8/ABwcvY4B/yXRcc1shTQxqdqmqmmrLlQ0JQn+EfEF4NKUTW4HPhOZp4F3Sro+xbHNbLUsKwdexziYl9XP5NNWXUkNtZXzvwF4Jff+wugzM7NeywfzIuO01fgGcddd3UgNdWrCV9IxSVuStra3t5d9OmZmMxXl+cfGaSuYfoOA9iuarmnpOK8CN+be7x99doWIOAGcgKzOv51TMzNbXFnQli6vTdjYKL9BjLVd0dTWyP8M8JFR1c+PAd+OiNdaOraZWWOqlKfOGtUvo6IpVannfwe+CPyQpAuS7pP0UUkfHW3yOPAycBb4r8DHUhzXzGzZqpSnThvVL6uiKUnaJyLunPHzAP5NimOZmXXJOGhPa9Fw/HiW88+nftbWllvG2lbO38xsZR09Oj2IV7lBtK1T1T5mZl1QZRFWX3r4lPHI38wsZ1y3P07RjBdhweWAXWWbrvPI38wGq2j0XqU/T596+JTxyN/MBqls9F5Wj58v11yF1tMe+ZvZyptnhL9zZ/E+8uWafW89DQ7+Zrbiihqp3X13eauF7353dt1+31tPg4O/ma24ohH+tKfXjhddTWsr3ffW0+Bn+JrZituxY3qwz0u98Cr183mr8DN8zcyonodPPXrvSt/+Mg7+ZrbSivLzk9bX0y+86no5qIO/mSXRhUcTFsnn5yHL0efNmqhd9Lq6Xg7q4G9mtXU9xTFurRABJ09Wn6itc11dLwd18Dez2rqS4qgySp+nx06d6+p6OaiDv5nV1oUURxN/fdS5rq6Xg7rU08xq29goXjQ1nkjt6zl04brm5VJPM2tNF1IcTfz10YXraoqDv5nV1lSKY55KmyYmWLueuqnDaR8z66TJrpswfQXuvNuvqlbTPpIOS3pR0llJDxT8/ICkz0v6M0nPSvpgiuOaWfuaqOefp+vmXXcVH3eVR+mNiIhaL2An8HXgB4BdwFeBmye2OQH8zOj7m4FvzNrvoUOHwsy65dSpiLW1iKyeJnutrWWfp95n/n3Rq+5xi85jfT1Cyr6m3HebgK2oELtTjPxvBc5GxMsR8QbwCHD75D0GePvo+3cAf5HguGbWsibq+eftq5/quHldX6TWhBTB/wbgldz7C6PP8n4JuEvSBeBx4N8W7UjSMUlbkra2t7cTnJqZpdRERU3Z7xb11U953LyuLFJrU1vVPncCvxMR+4EPAiclXXXsiDgREZsRsblv376WTs3Mqmqioqbsd/N99ef93Xl1YZFa21IE/1eBG3Pv948+y7sPeBQgIr4I/B1gb4Jjm1mLmqh7n7bPcSuGU6fKt0kxAd31PjxNSBH8nwEOSrpJ0i7gDuDMxDbngfcBSPr7ZMHfeR2znmmioqbKPsu2gTS5+lVezFWqyqzwrBdZKudrZFU/D44+ewg4EpcrfP6UrBLoK8A/nbVPV/uYta9vFS/r68WVQOvr8++rb9dehorVPl7kZWZAPxdJlT2iUcq6dg6Re/uY2Vz6WPEyxFx9Kg7+Zgb0s+JlkLn6RBz8zQzo5yjaLR0W5+BvZkC/RtH58s4HH8zOscqTuewyB38zA/ozih5iK4YmOPib2d+a5/m2dS26OKuPE9Nd5OBv1hFNtEruqjqj96oT00P691yEg79ZBwwtlVFn9F5lYnpo/56LcPA364BVS2XMGnXXKSutMjG9av+eTXDwN+uAPtbYl6ky6q5TVlplYnqV/j2b4uBv1gEpa+xT5bqbnJCtW1Y6a2K6j2sWWlelAdAyXm7sZkOS6vGITe5Hutw0bdr+xttNvqSrj9FUI7UmHjfZF1Rs7Lb0IF/2cvC3oUkRDFN1uSzbT5VAmrLTZh2r0qVzXlWDv7t6mq2QVF0uy/aTt76epVwm9bE76CpxV0+zAUqV666yfdnkaV9WCg+dg7/ZCknVn6doP5Om3SDKJmS98Ko7HPzNVkiqUXd+P5DtK2+RG4oXXnWLc/5mNtPp01mp5vnz2Yh//HD1eWxsZAF/UtncgS2m1Zy/pMOSXpR0VtIDJdt8WNLzkp6T9Lspjmtm7UjR8M0Lr7rlmro7kLQTeBh4P3ABeEbSmYh4PrfNQeAXgJ+IiG9J+rt1j2tm/XLgQPHI3wuvliPFyP9W4GxEvBwRbwCPALdPbPOvgYcj4lsAEfHNBMc1sx4YT/KeO5dm7sDSSBH8bwBeyb2/MPos713AuyT9qaSnJR1OcFyzQehzhUx+kheyid7xDcAloMtVO+0zx3EOAu8F9gNfkPQPIuL/5DeSdAw4BnDAfwuaXbVgalwhA/0ImkV9fiI8ydsFKUb+rwI35t7vH32WdwE4ExH/LyL+F/A1spvBFSLiRERsRsTmvn37EpyaWb/1vTWxJ3m7K0XwfwY4KOkmSbuAO4AzE9v8D7JRP5L2kqWBXk5wbLOVljJ4LiN95O6a3VU7+EfEm8D9wBPAC8CjEfGcpIckHRlt9gRwUdLzwOeBn4+Ii3WPbbbqqgbPWYG9qQVWs46basWxNaBK97dlvNzV06xaa+Iq20zrtLlo98uqbZOH2l1zWXBLZ7PVkA+ee/Zkr3wgrRLYZ7VnXqTvfVdaN9uVqgZ/t3cwa1GdNglFrZKl6a2X19aunjDO27kTvvvdqz+vUo2Tqn20peWWzmYdUzfvXlY2WWbnzumBf22tOPBDtQllT+b2m4O/WUvmKdssmkidp8JnWmCHywusxl07J1UJ4J7M7TcHf7OWVC3bLPsLYffuaseZFdjHKZ2jR+sFcD+0pd+c8zdrSdWWxmXb7dkD3/nO9FROfl/THqcIl+cexjeVS5cWb9ds3eGcv1nHVB1ll/2FcOnSfA9YKRuZw5V/WVy8mN1UTp5cvF2z9VCVkqBlvFzqaauoSs171RLKRevnXaK52nCpp1k/TUvXpBiVu0RztTntY9ZTTU+kukTTwMHfrJNSPDaxjEs0DRz8zQbHJZoG7T3Mxcw65OhRB/uh88jfrII+P0rRrIiDv1mJccCX4O675+vJ45uFdZ2Dvw1eUaAuevB43rRHKTb14BSzlFznb4NWVlN/7bXZytdpyuriq7ZxMGuC6/ytU7qaBinrtDkr8EN5XbwfWm59kCT4Szos6UVJZyU9MGW7n5IUkmbelWx1dDkNsmhALqqLH9/gyv6Y9iIq65LawV/STuBh4APAzcCdkm4u2O5twM8CX6p7TOuXefrYt60sIO/Zc/VCqHEjtaK6+Mk5gkleRGVdk2LkfytwNiJejog3gEeA2wu2+2Xgk8BfJzim9UiX0yBlq11//devXgh18mQ2qi9acVt0gxvzIirrohSLvG4AXsm9vwD8o/wGkt4N3BgRn5P08wmOaT1y4EDxiLgLaZBxQC57rm7VgF12I5M8yWvd1PiEr6QdwK8A/67CtsckbUna2t7ebvrUrCVd7yVTp4+O8/zWVymC/6vAjbn3+0efjb0N+BHgKUnfAH4MOFM06RsRJyJiMyI29+3bl+DUrAtWtZeM8/zWZ7Xr/CVdA3wNeB9Z0H8G+BcR8VzJ9k8BPxcRU4v4XedvXVdWzw/ZDc6PQ7RlaK3OPyLeBO4HngBeAB6NiOckPSTpSN39m3XVrDz/rMDf1bUPNgxJunpGxOPA4xOf/WLJtu9NcUyzZaszkT25sni89gH814K1wyt8zRZUZyK7y2sfbBgc/M0WVGciu8trH2wYHPzNcubNwy9aJurn6NqyOfibjbTZg6jrax9s9Tn4m420mYdf1bUP1h8O/mYjTeXhy1JJdVYWm9Xl4G820kQevsvtrG3YHPytdfmR8N692asLC52q5uHnmRR2Sad1lYO/tWpyJHzxYvZqY1Q8K2hXycPPO5J3Sad1lZ/ha62a1g9nrIln3ZY9q3feSdZ5n8/r5/la2/wMX+ukKiPeRUbFs0b1qdIv847kXdJpXeXgb62qMnl64MB8efUqqZhU6Zd5J4Vd0mld5eBvrSoaCeetrcEHPzhfXn3aqH7Ww1Yi5ptoXmQk75JO6yIHf6ssRQviyZHwnj3ZKz8qfvzx+VI0ZaP38U1j1hzDPBPNHsnbyoiITr4OHToUlt6pUxHr6xFS9vXUqeq/t7YWkY2Vs9faWvXfn+fc8sfIv6Ti35v2O/O81tfTXYvZsgBbUSHGeuQ/IHUWHDVdrz7rkYhQnleflUqqyuWXNiQO/gNSJ4A3Xa9edG550/Lq+VRMFTt3Fn/ujpo2JA7+PTdPHr5OAG+6BfG0c8jn1Wf1yZGmH2dtLfsLw+WXNnhVckPLeDnnP9u8efiy3HiVXHfTOf8q51blHKbl//NzHIvOfZh1HRVz/kkCNXAYeBE4CzxQ8POPA88DzwJPAuuz9ungP9u8wbxuAG8yYNYJ7PPeIMxWWWvBH9gJfB34AWAX8FXg5olt/gmwNvr+Z4DPztqvg/9sUnEwLKuKiej2iHfWuVW93i5fo1nTqgb/2r19JP048EsR8c9G739hlE76DyXb/yjwGxHxE9P2694+sw2tb8zQrtdsEW329rkBeCX3/sLoszL3AX9Y9ANJxyRtSdra3t5OcGqrbWh9Y4Z2vWZNarXaR9JdwCbwqaKfR8SJiNiMiM19+/a1eWq9NLTVpkO7XrMmXZNgH68CN+be7x99dgVJtwEPAu+JiL9JcFwjC3xDCn5Du16zpqQY+T8DHJR0k6RdwB3AmfwGozz/bwJHIuKbCY5pZmY11A7+EfEmcD/wBPAC8GhEPCfpIUlHRpt9Cvhe4PckfUXSmZLdmZlZC5Lk/CPi8Yh4V0T8vYg4PvrsFyPizOj72yLiuoi4ZfQ6Mn2PNilFR82+GvK1mzUlRc7fGjb5CMJxQzZY/fz3kK/drEl+hm8PDLm+fcjXbrYIP8N3hTTdUbOuJtMy067d6SCzxTn490DTHTXrqPOMgCrKrnH37maPa7bqHPx7oI2VrYuOopt+yEvZtY+P09RxzVadg38PNL2ytc7ovemUVNm1X7rU7HHNVp0nfK3WpGrZ745///jxZqpyPBFsVswTvlZZnUnVac/PPXcO7r47G7GnnpB1kzezehz8eyhVlct4P2V//FWZVJ31/NzxvlNPyLrJm1k9Tvv0zOSiJ8hGvPMGvqL95K2twbXXwsWLV/+sLLWyY0f5jWTW75pZGk77rKhU1TVF+xlbdFK1SumpJ2TNusHBv2dSVdeUbS9lI/OjR+dfXzAt/z/rd82sXQ7+PZNqwVeV/cw7qTqZ/5eq/66ZtcvBv2dSVblU2c+0SdWySeejR7O/HCLg5ElPyJp1VpWnvC/jdejQoYWeXD8Ep05FrK9HSNnXU6fa3c+pUxFraxFZiM9ea2uLn4eZpQNsRYUY62qfDjh9OpuAPX8+S7s0tTAqFS+wMusuV/v0RFFrhaYWRqXS9S6jZjabg/+SFZVcNrUwKpUudxk1s2qSBH9JhyW9KOmspAcKfv49kj47+vmXJG2kOO4qmDVa7mKnSrdWMOu/2sFf0k7gYeADwM3AnZJuntjsPuBbEfGDwK8Cn6x73FXRxYVRs9pHuLWCWf+lGPnfCpyNiJcj4g3gEeD2iW1uBz49+v4x4H3SZBX4MHVtYVTV9s7jks633rq8KMzM+iNF8L8BeCX3/sLos8JtIuJN4NvAngTH7r2uLYxq+uEsZtYNnZrwlXRM0pakre3t7WWfTmu6tDDKlTxmw5Ai+L8K3Jh7v3/0WeE2kq4B3gFc1S8yIk5ExGZEbO7bty/BqfXPstMpruQxG4YUwf8Z4KCkmyTtAu4Azkxscwa4Z/T9h4A/ia6uLksgVb/9ZXAlj9kw1A7+oxz+/cATwAvAoxHxnKSHJB0ZbfbbwB5JZ4GPA1eVg66KOs/DTX0ei9yAXMljNgxu75BYF1ofpHrgi5n1j9s7LEkXJkxdsWNmszj4J9aFCdMu3IDMrNsc/BPrwoRpF25AZtZtDv6JpZwwXXTStgs3IDPrtkEH/6ZKMlPU6tepGnLFjpnNMtjg35WSzDJ1J23zN6Djx7Pf6+O6AzNrxmBLPbtQkjnNjh2X+/rnSVlAr8pln2bD4lLPnKL0TtcrYlJN2rrs08yKrHzwL0vv7N5dvH1XKmJSTdp2/SZnZsux8sG/bOQL3a6ISTVp67JPMyuy8sG/bIR76VL3K2JSVA257NPMiqx88J828m27fXLT3T6L9u+yTzMrcs2yT6Bpx48XV7u0PfKdrLoZzz1AmkA8a/8O9maWt/Ij/y6suIXmq25c1WNm8xhsnf+86tbLp6rbX9b+zawfXOefWN2RddNVN67qMbN5OPhXVLdevqjqRspy8ykmf13VY2bzcPCvqO7IOj/3AFngH6dpUvQVclWPmc2jVvCXtFvSH0l6afT1+wq2uUXSFyU9J+lZST9d55htG0/ynjuXBdW8eUfW49LS9fWr8/MpJmfbLl01s/6qO/J/AHgyIg4CT1L8YPbXgY9ExA8Dh4Ffk/TOmsdtRb41BGQBe3wDqDqy7mNfITNbfbWqfSS9CLw3Il6TdD3wVET80Izf+SrwoYh4adp2Xaj2qdv5s6xC6Npr4eLFxfdrZlamrWqf6yLitdH3fwlcN+OkbgV2AV+vedxW1B2h97WvkJmtvpnBX9IfS/rzgtft+e0i+xOi9M+I0V8GJ4F/GRGFleeSjknakrS1vb0956WkN2uSd9airz73FTKz1dZK2kfS24GngH8fEY9V2XfbaZ/Tp7OR+vnzWXAfj8LLFnZN+9k4iHf9gTFmtnraSvucAe4ZfX8P8AcFJ7IL+H3gM1UDf9uKev7ffTfcdVeWn9+z5+oRepVFX669N7Ouqhv8PwG8X9JLwG2j90jalPRbo20+DPwkcK+kr4xet9Q8blJFgXz8B9HFi/Cd78DJk1eWT1aZD3DtvZl1lXv7UN4XJ28yVeOUjpl1kXv7zKHKKt3Jkb5TOmbWZw7+FAfySZM3CKd0zKzPVv5hLlWMA/aDD15u45BPA5WN6P2QFDPrK4/8R8Z9cSKyyV2P6M1slTn4F1ikQVrTz+c1M0vJaZ8Emn4+r5lZah75J+Dn55pZ3zj4J+AWzWbWNysb/NvMwfv5uWbWNysZ/It69dR9TOI0XvBlZn2zksG/7Ry8F3yZWd+sZPCfloNvKh3k5+eaWZ+sZPAvy7Xv3t1uOsjMrKtWMviX5eDBJZlmZrCiwb8sB3/pUvH2Lsk0s6FZyeAPxTl4l2SamWVWNvgXcUmmmVlmUMHfJZlmZpnBNXZzD34zs5ojf0m7Jf2RpJdGX79vyrZvl3RB0m/UOaaZmdVXN+3zAPBkRBwEnhy9L/PLwBdqHs/MzBKoG/xvBz49+v7TwD8v2kjSIeA64H/WPJ6ZmSVQN/hfFxGvjb7/S7IAfwVJO4D/BPzcrJ1JOiZpS9LW9vZ2zVMzM7MyMyd8Jf0x8P0FP7piXWxEhKQo2O5jwOMRcUHS1GNFxAngBMDm5mbRvszMLIGZwT8ibiv7maT/Len6iHhN0vXANws2+3HgH0v6GPC9wC5J/zcips0PmJlZg+qWep4B7gE+Mfr6B5MbRMTfFlZKuhfYdOA3M1suRSyeXZG0B3gUOACcAz4cEZckbQIfjYh/NbH9vWTB//4K+94e7bOv9gJ/teyTaJGvd7UN6Xr7fq3rEbFv1ka1gr+Vk7QVEZvLPo+2+HpX25CudyjXOqj2DmZmlnHwNzMbIAf/5pxY9gm0zNe72oZ0vYO4Vuf8zcwGyCN/M7MBcvBPZGgdTqtcr6RbJH1R0nOSnpX008s41zokHZb0oqSzkq5anyLpeyR9dvTzL0naaP8s06hwrR+X9Pzov+WTktaXcZ6pzLre3HY/JSlGJewrw8E/naF1OK1yva8DH4mIHwYOA78m6Z0tnmMtknYCDwMfAG4G7pR088Rm9wHfiogfBH4V+GS7Z5lGxWv9M7J1Ov8QeAz4j+2eZToVrxdJbwN+FvhSu2fYPAf/dIbW4XTm9UbE1yLipdH3f0HW/mPm4pMOuRU4GxEvR8QbwCNk152X/3d4DHifZjWx6qaZ1xoRn4+I10dvnwb2t3yOKVX5bwvZQO2TwF+3eXJtcPBPJ2mH0x6Yeb15km4FdgFfb/rEEroBeCX3/sLos8JtIuJN4NvAnlbOLq0q15p3H/CHjZ5Rs2Zer6R3AzdGxOfaPLG2DO4xjnW02eG0CxJc73g/1wMngXsi4q20Z2ltk3QXsAm8Z9nn0pTRQO1XgHuXfCqNcfCfw9A6nCa4XiS9Hfgc8GBEPN3QqTblVeDG3Pv9o8+Ktrkg6RrgHcDFdk4vqSrXiqTbyG7+74mIv2np3Jow63rfBvwI8NRooPb9wBlJRyJiq7WzbJDTPumMO5zClA6nEXEgIjbIUj+f6Wrgr2Dm9UraBfw+2XU+1uK5pfIMcFDSTaNruYPsuvPy/w4fAv4k+rl4Zua1SvpR4DeBIxFReLPvkanXGxHfjoi9EbEx+v/1abLrXonADw7+KX0CeL+kl4DbRu+RtCnpt5Z6Zs2ocr0fBn4SuFfSV0avW5ZzuvMb5fDvB54AXgAejYjnJD0k6chos98G9kg6C3yc6VVenVXxWj9F9hfr743+W07eCHuj4vWuNK/wNTMbII/8zcwGyMHfzGyAHPzNzAbIwd/MbIAc/M3MBsjB38xsgBz8zcwGyMHfzGyA/j9ZHUfMtzg+PgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45843017f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# first generate some observations\n",
    "true_a = 1.3\n",
    "true_b = 0.4\n",
    "data_x = (np.arange(100) / 99.0 - .5).astype(np.float32) # Chainer assumes all the cpu computation is done in float32\n",
    "data_y = (data_x * true_a + true_b + np.random.randn(*data_x.shape) * 0.1).astype(np.float32)\n",
    "_ = plt.scatter(data_x, data_y, c='b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chainer provides an abstraction called `Link` that describe some computation and keeps track of parameters for it. For instance, a `Linear` link describes a linear map on input and keeps track of `w` and bias `b`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w: variable W([[ 0.54163736]])\n",
      "b: variable b([ 0.])\n",
      "Operations on chainer.Variable: variable([[ 123.54163361]]), type: <class 'chainer.variable.Variable'>\n",
      "numpy arrays: [[ 0.54163736]] [[ 123.54163361]]\n"
     ]
    }
   ],
   "source": [
    "model = L.Linear(in_size=1, out_size=1) # input is 1D data and output is also 1D data\n",
    "\n",
    "# Chainer will randomly initialize `w` and `b` for us.\n",
    "# we can take a look at their values\n",
    "print(\"w:\", model.W)\n",
    "print(\"b:\", model.b)\n",
    "\n",
    "# model.W and model.b have type `chainer.Variable`, \n",
    "#   which is a wrapper around Numpy array\n",
    "assert isinstance(model.W, chainer.Variable)\n",
    "\n",
    "# operations that involve `chainer.Variable` will produce\n",
    "#   `chainer.Variable` and this records the computation graph\n",
    "var_result = model.W + 123 # some random computation\n",
    "print(\"Operations on chainer.Variable: %s, type: %s\" % (var_result, type(var_result)))\n",
    "\n",
    "# the underlying numpy array can be accessed by `data` attribute\n",
    "print(\"numpy arrays:\", model.W.data, var_result.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2wZHV95/H3Zy4MM1eGGZgZEZiZO2wyWkE3RWSCa7lZzYIRqV3YKi2FHRC2IFPqks2WSVYsajcu7qwas4lawY2jJiIzuy5qmcxGXKIIZZIFl2tJ1MFCRmAeeJDLPIkC8vTdP0535kxPn+7TfU6fPt3n86rqmn44fR4uxff369/v+/seRQRmZtYsi8Z9AmZmVj0HfzOzBnLwNzNrIAd/M7MGcvA3M2sgB38zswZy8LepIumrkq7o8fmfSvqPOfd1h6Sryzu7fCS9QdK+nNu+X9K2UZ+TTZ/jxn0CZv1Iegi4OiK+3m/biHhz6ntXtr73T1Ofv3MU52g2adzzNzNrIAd/myiSrpT0t5L+UNJBSQ9KSvf275B0taRfAv4UeK2kn0o61Pr8s5L+S+v5yZL+StJCa19/JWlNzvN4v6QvSNom6UlJ35P0cknvk/S4pL2SfiO1/emSdkg6IGmXpN9Mfba0dV4HJd0L/GrHsU6X9KXWeT4o6d8V+iOa4eBvk+k1wH3AKuAPgM9IUnqDiPgB8E7gzog4MSJWdNnPIuDPgTlgHfA08CcDnMe/BG4CTga+A9za2ucZwPXAJ1Pbfh7YB5wOvBX4r5L+eeuz3wd+ofV4E/APcxaSFgH/G/j71n7PA/69pDcNcJ5mx3Dwt0m0OyI+FREvADcCpwGnDrqTiNgfEV+KiKci4klgC/D6AXbxNxFxa0Q8D3wBWA18KCKeIwn26yWtkLQWeB3w3oh4JiLuAT4NvKO1n7cBWyLiQETsBT6eOsavAqsj4vqIeDYiHgA+BVwy6PWapXnC1ybRY+0nEfFUq9N/4qA7kTQL/DFwAUnvHWCZpJlWw9LPj1PPnwaeSH3v6da/J5L09g+0Gpi23cDG1vPTgb0dn7XNAae3h61aZoC/yXF+Zpkc/G2a9StZ+zvAK4DXRMRjks4mGb5R768N7BHgFEnLUg3AOuDh1vNHgbXAztRnbXuBByNiQ8nnZA3nYR+bZj8G1khanPH5MpIe+iFJp5CMvZeuNZTzf4EPSloi6ZeBq4B2fv7NwPtaE9BrgN9Kff3/AU9Kem9rYnhG0qskHTUpbDYoB3+bZt8g6U0/JumJLp9/FFgKPAHcBfyfEZ7LpcB6kl8BXwZ+P7Vu4T+TDPU8CPw1ySQyAK1hpH8BnN36/AmS+YLlIzxXawD5Zi5mZs3jnr+ZWQM5+JuZNZCDv5lZAzn4m5k1UG3z/FetWhXr168f92mYmU2Ub3/7209ExOp+29U2+K9fv575+flxn4aZ2USRtLv/Vh72MTNrJAd/M7MGcvA3M2sgB38zswZy8DczayAHfzOzBnLwNzPLaft2WL8eFi1K/t2+fdxnNDwHfzOzHLZvh82bYfduiEj+3by5eAMwrgbFwd/MLIfrroOnnjr6vaeeSt4f1qgalDwc/M3MctizZ7D38xhFg5KXg7+Z1Updx9XXrRvs/TxG0aDk5eBvZrUxzmGQfrZsgdnZo9+bnU3eH9YoGpS8HPzNrDbGOQzSz6ZNsHUrzM2BlPy7dWvy/rBG0aDkVdt7+G7cuDFc1dOsWRYtSnr8nSR48cXqz6cK27cnjduePUmPf8uWYg2KpG9HxMZ+27nnb2a1Mc5hkLIMOmexaRM89FDSuD30ULHAPwgHfzOrjXEOg5ShznMWnUoJ/pL+TNLjkr6f8bkkfVzSLknflfTqMo5rZtNlFOPqVarznEWnsnr+nwUu6PH5m4ENrcdm4L+XdFwzmzLjGgYpQ97UzTqks5YS/CPim8CBHptcDHwuEncBKySdVsaxzczGrR3Ms/Jn1q07so0El18+/qGhqsb8zwD2pl7va71nZjbR0uP83czOwoUXHr1NZyMxjqGhWk34StosaV7S/MLCwrhPx8ysr27j/G3tOYtbbsnepq2KVb1pVQX/h4G1qddrWu8dJSK2RsTGiNi4evXqik7NzGx4WUFbOjJnkSewV53OWlXw3wG8o5X180+AwxHxaEXHNjMbmTxrE/oF9nGks5aV6vk/gTuBV0jaJ+kqSe+U9M7WJrcADwC7gE8B7y7juGZm45ZnbUK3baTk33Glsx5Xxk4i4tI+nwfwb8s4lplZnbSDdq8SDXm2qVopwd/MrGkGrcmzaVO91izUKtvHzKwO+i3CmqQyDlkc/M3MUvIE9kkq45DFwd/MGqtbDz9PYB/nHbjK4jF/M2ukdg+/HejbPfysxVjpwL5uXfcVvZNUeto9fzObeuke/qpVyeOyy7r38Gdmuu8jHdgnvfQ0OPib2ZTrHMPfvz95ZHnhhf6BfdJLT4ODv5lNuV61d7ppB/J+gX2SS0+Dg7+ZTblBJmHbPfyyAnsd6vZncfA3s6mWdxK27KGbuq8FcPA3s6nWbXI2bXYWtm0rf+im7msBHPzNrBR1HeLonJxduTJ55J2oHfa66r4WwHn+ZlZYVs481GMidNi6OkWuq+5rAdzzN7PC6jLEUfavjyLXVfe1AA7+ZlZYHYY4RjHBWuS66r4WQJF1u/kx27hxY8zPz4/7NMwsh/Xruw9xzM0lE6mTeg51uK5BSfp2RGzst517/mZWWB2GOEbx66MO1zUqDv5mVtiohjgGGcPPcy/dQdV96KYID/uYWS11ZtpA0uvOCr6Dbj+tKh32kXSBpPsk7ZJ0bZfP10m6XdJ3JH1X0oVlHNfMqjeKfP5B6upfdln3405zL30kIqLQA5gBfgT8I2Ax8PfAWR3bbAXe1Xp+FvBQv/2ec845YWb1sm1bxOxsRJJPkzxmZ5P3y95n+nW3R9HjdjuPubkIKfm3zH1XCZiPHLG7jJ7/ucCuiHggIp4FPg9c3NnGACe1ni8HHinhuGZWsVHk82ftM6uuflnHTat7HZ5RKCP4nwHsTb3e13ov7f3AZZL2AbcAv9VtR5I2S5qXNL+wsFDCqZlZmUaRUZP13W519cs8blpdFqlVqapsn0uBz0bEGuBC4CZJxxw7IrZGxMaI2Lh69eqKTs3M8hpFRk3Wd9N19Qf97qDqsEitamUE/4eBtanXa1rvpV0F3AwQEXcCS4BVJRzbzCo0irz3Xvts19Xfti17mzImoEfRqNVdGcH/bmCDpDMlLQYuAXZ0bLMHOA9A0i+RBH+P65hNmFFk1OTZZ9Y2UM5Y/TQv5sqUZ1a434NkKOeHJFk/17Xeux64KI5k+PwdSSbQPcBv9Nuns33MqjdpGS9zc90zgebmBt/XpF17FnJm+3iRl5kBk7lIatGiJNx3kpJbMDaRa/uY2UAmMeOliWP1ZXHwNzNgMjNeGjlWXxIHfzMDJrMX7ZIOw3PwNzNgcnvR7XTQF18s/ybs08zB38yAyepF1/Vm8ZPEwd/M/kGVvehhA3gT6/CMgoO/WQ00rSdbJIDnzUpq2t90UM7zNxuzScyvL6rIvXHz5PY38W/a5jx/swkxifn1/fTrdRdJK82TlTSNf9OyOfibjdkk5tf3kmdIp0haaZ6spGn7m46Cg7/ZmJWdX1/WWPew+8nT6y6SVponK2kS1yxULk8BoHE8XNjNmqLMWyOWta8i+5G6F1uTjj3GqAqpjeJ2k5OCnIXdxh7ksx4O/tYkZQXCsqpcZu2nva9e51dmpc0ipqVK56DyBn9n+5hNkbKqXGbtp61X5kyTM23qwNk+Zg1U1lh3v+17Zc5M0krhJnPwN5siZdXn6bafTr0yZ1xvp/4c/M2mSFm97vR+sgyTOeNVt/XhMX8z66msMXzPBVTDY/5mVoqyfk141W29lBL8JV0g6T5JuyRdm7HN2yTdK2mnpP9RxnHNrBpljOF71W29HFd0B5JmgBuANwL7gLsl7YiIe1PbbADeB7wuIg5KemnR45rZZFm3rnsxN6+6HY8yev7nArsi4oGIeBb4PHBxxza/CdwQEQcBIuLxEo5rZhOgPcm7e3cybJQ2CXcKm1ZlBP8zgL2p1/ta76W9HHi5pL+TdJekC7rtSNJmSfOS5hcWFko4NbPJNunZMekib5AsHGs3AM7/H6/Cwz4DHGcD8AZgDfBNSf84Ig6lN4qIrcBWSLJ9Kjo3s1rqzI5pV8eEyQmY3SZ5I/LV7bfRKqPn/zCwNvV6Teu9tH3Ajoh4LiIeBH5I0hiYWYays2PG8SvCk7z1VUbwvxvYIOlMSYuBS4AdHdv8BUmvH0mrSIaBHijh2GZTq8zAOa773rq0cn0VDv4R8TxwDXAr8APg5ojYKel6SRe1NrsV2C/pXuB24PciYn/RY5tNs0ECZ79e/ahy7Psdt6xyEzYCeUp/juPhks7WdHlr0ufZrleN/WFLHw9yfk0srTwuuJ6/Wf0MGgg7t3/Xu479fq/6+b0+h4iVK4e/6Uld6vbb0fIGf9f2MatI0do23b4v9a+73znck/5s6VLY32UANk82Tln3DrByubaPWc0UHXfPSpvMMjOTHfjbOfYHDnT/PM+ksidzJ5uDv1lF8mbvZE2iDpLlMzsLL7zQ/TPpSH2eIgHck7mTzcHfrCJ5Am2vlMy8Pep2rz6rFn96P0UCuO/YNeHyTAyM4+EJX5s2ebJj+k3edn6/12Rrr+OlJ5JXrkwezsaZDuSc8HXP36wieXrKvYaGOu+u1a9IWtbx4OhfF/v3w9NPw003+ZaLTeJsH7MaaVe/7NQt+2b79mQSeM+eZChny5Z8gXuQY9jkcbaP2QQaZAx+2BusuN6OgYO/Wa1UMYnqFE0DB3+z2injlom9OEXTwMHfrHGcomlQ3c1czKxGNm1ysG869/zNcpj02ymadXLwt8bLCuzt9yW4/PLBboTixsLqznn+1mhZlTavuAJuvDG7MBpk58UXrd5pVkTePH8Hf2u0rAVPMzPZhdHaskoXexGVjZMXeVmt1HUYJGthU7/AD9l58V5EZZPAwd9Gblw3D88jK4DPzPT+Xre8+HYDl/Vj2ouorE5KCf6SLpB0n6Rdkq7tsd1bJIWkvj9JbHqM6ubhZcha8LR587HvtwupdcuLTzdw3XgRldVN4eAvaQa4AXgzcBZwqaSzumy3DPht4FtFj2mTpc7DIFkLnj7xiWPfv+mmpFffbdVttwauzYuorI4KT/hKei3w/oh4U+v1+wAi4oMd230U+Brwe8DvRkTP2VxP+E6PJkyA+n62VhdVTvieAexNvd7Xei99Mq8G1kbEV3rtSNJmSfOS5hcWFko4NauDaa4l43F+m1Qjn/CVtAj4I+B3+m0bEVsjYmNEbFy9evWoT80qMq21ZDzOb5OsjNo+DwNrU6/XtN5rWwa8CrhDyYzZy4Adki7qN/Rj02Maa8n0G+fPe3MVs3Eoo+d/N7BB0pmSFgOXADvaH0bE4YhYFRHrI2I9cBfgwG8TL2vCWspXirmuax+sGQoH/4h4HrgGuBX4AXBzROyUdL2ki4ru36yuitwUpc5rH6wZXN7BbEhFavg0IQPKxsPlHcxGrMhEdp3XPlgzOPibpQw6Dj/sLRd9H10bNwd/s5Yqx+Gnee2DTQYHf7OWKmsQTevaB5scDv5mLaMah88aShp2yMisDA7+Zi2jGId3SqfVlYO/VS7dE161KnnUYaHTKMbh61zO2prNwd8q1dkT3r8/eVTRK+6XyZN3HH6QjCCndFpdeZGXVSprcVPaKBY6lXVT9UH348VcVjUv8rJaytPjHaZX3K83Xtbwy6D7cUqn1ZWDv1Uqz+TpunWDDa3kmVQta/hl0P04pdPqysHfKtWtJ5w2OwsXXjhYhkye3nhWoxMx2ETzMBlBTum0OnLwt9zKKEHc2RNeuTJ5pHvFt9wy2NBKr954+5x37z5yA/ZOg0w0exjHpkZE1PJxzjnnhJVv27aIubkIKfl327b835udjUj6ysljdjb/9wc5t/Qx0g+p+/eyvrNy5bHnLGXvf25usPMc9G9oVgVgPnLEWGf7NEidSxB3O7e8x+r2XSn7vrpZfLN1mwbO9rFjFMl4GXW+eq9bIkLvoZX0UBIMF/jBFTWtWRz8J1xVC45GXYK41znkyZBpT6rOzfUP/CtXetzezMF/gg1aN6ZIAB/1RGfWObSHetqBv19j168hm52Fj33M6ZdmY5/YzXp4wre/rInOrInLopO2o5zozHNuebbpNWHsyVlrAnJO+JYSqIELgPuAXcC1XT5/D3Av8F3gNmCu3z4d/PvLylzJyoqJqHemSr9zy9PYVZGVZFZneYN/4WwfSTPAD4E3AvuAu4FLI+Le1Da/DnwrIp6S9C7gDRHx9l77dbZPf02rG7NoUffx/M4sne3bkwnkPXuS4aQtWzykY81RZbbPucCuiHggIp4FPg9cnN4gIm6PiHYux13AmhKO23hNW3CUd87CK2rN+isj+J8B7E293td6L8tVwFe7fSBps6R5SfMLCwslnNp0a1rdmKY1dmajdFyVB5N0GbAReH23zyNiK7AVkmGfCk9tYm3aNL3BvlP7Oj2kY1ZcGcH/YWBt6vWa1ntHkXQ+cB3w+oj4eQnHtQZqUmNnNkplDPvcDWyQdKakxcAlwI70BpJ+BfgkcFFEPF7CMc3MrIDCwT8ingeuAW4FfgDcHBE7JV0v6aLWZh8BTgS+IOkeSTsydmdmZhUoZYVvRNwSES+PiF+IiC2t9/5TROxoPT8/Ik6NiLNbj4t679E6lVFOeVI1+drNRqXSCV8bTmfVynYZB5j+8e8mX7vZKLmk8wRo2mKutCZfu9kwXNJ5ioy6nHJRoxyWyXOXLg8HmQ3OwX8CjLqcMgwfSAetLDqorGs85ZTRHtds2jn4T4BRr2wtEsCL3CAmj6xrbx9nVMc1m3YO/hNg1GUcRnGHr927yxmKybr2AwcGOx8zO5qD/4QYZbGyIuPqvYaedu+Gyy9PgnaRhqDbtVcxFGY2zRz8J1BZE53t/WQlfOUZV+82LJPW3nfZY/Iu8mZWUJ6i/+N4+GYu3ZV1s5Ju++nc58qV/W+e0t5Xrzto5bnL2DDqfGMas3Ghqpu5jIrz/LsrK+89az/tfW3ZkgzZ5Ll5Sp599vuumZXDef5Tqqyc/6ztpeHH1fsNAfX6rplVa7rLOxw8CB/4AKxYkTxOPvnI8/TjxBOTqDcB1q3r3rseNKjm2c+WLUeXVoDe4+rpevu7dyd/0vQvB4/Jm9XHdAf/J56AT30KfvrT3tvNzMDy5cc2ClmNRefjJS+prPEYNCAX2c8wN09J19v3vXTN6qsZY/7PPQc/+UnyS+DQoeRx8CAcPpzv+c9+1nv/MzPdG4vOBiWrMZmdHajxKCuoFtmPA7tZPeUd829G8C/queeSxuDw4aRBGKThOHgQnn669/6PO+5I45BuMPr98mh/vmRJpcNWnZU2IWm/pvn+wWaTwsG/Tn7+854NxPf/9hD33H6QRT89zOlLDvKqMw6y6vjUNj/vc9fLxYuTRiNvY9H5WLJkoMtxpU2z+sob/Kd7zL8uTjgBXvrS5NFh+3bYfFuqF/0M6IFkonRuDrZ8Bja95ZkjvyLy/OI4dCiJzu0hrn6Nxwkn5JvbaD1O272CEziZQ6zgECt4lhMAl1YwmyQO/mPWra5O56pYWMKmTUvg1FOHO8gzz2QPSaXfT8+HPPDAkW2ee+6o3d3ZsfunWcIhVvCzmRXw2gF/daxYkfxyMbNKlTLsI+kC4GPADPDpiPhQx+cnAJ8DzgH2A2+PiId67XOqhn16WLQou7xC21iHUyKSOYtUY3H7Xxzmcx8/xOyzB1nOYVZwiFUzhzjvnIPMLe/SwDz/fO9jLF06+HxH+nH88dX8LcwmQGXDPpJmgBuANwL7gLsl7YiIe1ObXQUcjIhflHQJ8GHg7UWPPQ2y8u3Tqh5OOTqTR2zZMsumTbNw2mkA/Prr4JGzj832mes22RuR/LTp/KXRft0t++rxx+H++498/sILvU94dnb4NN3ly914WCMV7vlLei3w/oh4U+v1+wAi4oOpbW5tbXOnpOOAx4DV0ePgTen5d8uc6VRlz792mTwRSaptr8ai3zBWv3oSL3nJYL800o3L8uVJqq9ZTVQ54XsGsDf1eh/wmqxtIuJ5SYeBlcATJRx/otVtVWyv2v5jCf5SsgL7xBPhjDMG/35Essgvb2Nx+DA88gjs3Hnkdb/GY9my4RqOFSvgpJPceNhY1GrCV9JmYDPAugYVganTqti63y94YFISnJctG+77L74ITz7ZO7Oq81fJnj3wve8d+azfr+uTThqswUg/TjopmTgyG1AZwf9hYG3q9ZrWe9222dca9llOMvF7lIjYCmyFZNinhHObOOmGYBzKqh00NRYtSoZ2li8f7o/QbjyyGo5u7+/eDffckzx/8sne+5eObTwGGcJatsyNR0OVEfzvBjZIOpMkyF8C/OuObXYAV5BkCb4V+Eav8f5JN+7eexFl1Q6ylnTjMTc3+PdfeCEpTdL+FdGv4Th8OEnTbT//yU9671/KX4Ykq/GYkKKIdrTCwb81hn8NcCtJquefRcROSdeT3FRgB/AZ4CZJu4ADJA3EVOqcMD2Sq19tAzBsAzRMMTcboZmZJBiffPJw33/hhaPnNPIsFLz//iPb9yuKuGjRYPMdnY1LhUUR7Wgu71CyOpQ+qF3Gjk2udlHEPKvKu73fK40NjhRFHHS+o/1rxY3HMVzbZ0yyFm1VeQerOjRAZkDSeOQpTZJVMLFf49Euipg3LbezMVm6dOoaD9f2GZM6TJhOXcaOTa7jj4dVq5LHMJ599tj5jn7VdfftO/K8X0Xd448frLHofCxdOtx11YCDf8nqMGFahwbIrBSLF2cWRcylXVF3kFLse/fmr6g7YFHEYx4DVtQtk4N/ycqcMB120rYODZBZLfSoqJvLM88c20B0FkHsnPt48MEjzzuKIh5jyZIjvzBe+Ur40peGO88hND74jyIts4xc/SJZQ87YMSvJkiXwspclj0FFHFtRt9ecx8qV5Z9/D42e8K1zVkyZk7aTvO7AzAbjbJ8c6pwVU1bWUJ0bODMrX97g35h13du3J8F+0aLk3+3b650VkzU5O+ikba9CbWbWXI0I/u3e7+7dSW+6PX5+yindt69DVsyWLUkPPW2YSds6N3BmNj6NCP5ZvV8oJ8COwqZNydDM3Fwy1DM3N9xQTVm/IMxsujQi+Gf1cg8cKCfAjsqmTcncw4svJv8Oc15l/YIws+nSiODfq/dbRoDNq9u8w6j3X9YvCDObLo3I86/DoqdRV/vst38HezNLa0yq57hz3UedVlrntFUzq45TPTuUNbwz7NDNqLNunNVjZoNoTPAvQ1bKaJ4GYNRZN87qMbNBOPgPoMiCqW5ZN1LSgJQx+eusHjMbhIP/AIoMraSzbiAJ/O3plkF+QeTZv7N6zKyfxkz4FtGeLO42oQqDT6p6ctbMRqWSCV9Jp0j6mqT7W/8ec5dpSWdLulPSTknflfT2IsesWnqcv5t+QyuTVlPIzJqh6LDPtcBtEbEBuK31utNTwDsi4pXABcBHJa0oeNzKdBvnb+s3tDKJNYXMrBmKLvK6GHhD6/mNwB3Ae9MbRMQPU88fkfQ4sBo4VPDYlcjqjUv9h2iyJoiXLk1+MfhOW2Y2LkV7/qdGxKOt548Bp/baWNK5wGLgRxmfb5Y0L2l+YWGh4KmVo0gK5aTWFDKz6dc3+Ev6uqTvd3lcnN4ukpnjzNljSacBNwH/JiK63o4kIrZGxMaI2Lh69eoBL6W4buPzvVIo+y34qktNITOzTn2HfSLi/KzPJP1Y0mkR8WgruD+esd1JwFeA6yLirqHPdoS61ca5/PJkrH7lymSo5sCBI6UhoH+tnjrUFDIz66bosM8O4IrW8yuAv+zcQNJi4MvA5yLiiwWPNzLdxufbWbD798PTT8NNNx3ppedZ8OXcezOrq0J5/pJWAjcD64DdwNsi4oCkjcA7I+JqSZcBfw7sTH31yoi4p9e+q87zz7pnblo6D7+se+yamZUpb55/oWyfiNgPnNfl/Xng6tbzbcC2Isepwrp12bn8bekJ3Kztna5pZpPA5R1auk3sdkoHdtfSMbNJ5uDf0q32TlpnYPd4vplNMtf2yTDum7+YmQ2jkjH/aeZbH5rZNPOwT0lGfXN2M7MyuedfglHfnN3MrGzu+ZegyB2+zMzGYaqDf1VDMa7Pb2aTZmqDf5GbrQ/KN083s0kztcG/yqEYL/gys0kztcG/yqEYL/gys0kztcG/11DMKOYCXJ/fzCbJ1Ab/rKGYCy+sbi7AzKyupjb4Zw3F3HKL0zLNzBpX28d1+M1smuWt7TO1Pf8sTss0M2tg8HdapplZA4O/0zLNzBpa2M3lms2s6Qr1/CWdIulrku5v/Xtyj21PkrRP0p8UOaaZmRVXdNjnWuC2iNgA3NZ6neUDwDcLHs/MzEpQNPhfDNzYen4j8K+6bSTpHOBU4K8LHs/MzEpQNPifGhGPtp4/RhLgjyJpEfDfgN/ttzNJmyXNS5pfWFgoeGpmZpal74SvpK8DL+vy0VFrYiMiJHVbMfZu4JaI2Cep57EiYiuwFZJFXv3OzczMhlNoha+k+4A3RMSjkk4D7oiIV3Rssx34NeBF4ERgMfCJiOg1P4CkBWD30Cc3fquAJ8Z9EhXy9U63Jl3vpF/rXESs7rdR0eD/EWB/RHxI0rXAKRHxH3psfyWwMSKuGfqgE0LSfJ4l1tPC1zvdmnS9TbnWomP+HwLeKOl+4PzWayRtlPTpoidnZmajUWiRV0TsB87r8v48cHWX9z8LfLbIMc3MrLjGlXeo0NZxn0DFfL3TrUnX24hrrW1JZzMzGx33/M3MGsjB38ysgRz8S9SkQnd5rlXS2ZLulLRT0nclvX0c51qEpAsk3SdpVyudufPzEyT9r9bn35K0vvqzLEeOa32PpHtb/y1vkzQ3jvMsS7/rTW33FkkhaarSPx38y9WkQnd5rvUp4B0R8UrgAuCjklZUeI6FSJoBbgDeDJwFXCrprI7NrgIORsQvAn8MfLjasyxHzmv9Dsk6nV8Gvgj8QbVnWZ6c14vwOT/iAAACLklEQVSkZcBvA9+q9gxHz8G/XE0qdNf3WiPihxFxf+v5I8DjQN+VhzVyLrArIh6IiGeBz5Ncd1r67/BF4Dz1q2NST32vNSJuj4inWi/vAtZUfI5lyvPfFpJO2oeBZ6o8uSo4+Jer1EJ3Ndf3WtMknUtS2uNHoz6xEp0B7E293td6r+s2EfE8cBhYWcnZlSvPtaZdBXx1pGc0Wn2vV9KrgbUR8ZUqT6wqjbyTVxFVFrobtxKutb2f04CbgCsi4sVyz9KqJukyYCPw+nGfy6i0Oml/BFw55lMZGQf/AUXE+VmfSfqxpNNShe4e77LZa4Ffk/RuWoXuJP20X6G7cSjhWpF0EvAV4LqIuGtEpzoqDwNrU6/XtN7rts0+SccBy4H91ZxeqfJcK5LOJ2n8Xx8RP6/o3Eah3/UuA14F3NHqpL0M2CHpolYFg4nnYZ9y7QCuaD2/AvjLzg0iYlNErIuI9SRDP5+rY+DPoe+1SloMfJnkGr9Y4bmV5W5gg6QzW9dyCcl1p6X/Dm8FvhGTuXKy77VK+hXgk8BFEdG1sZ8gPa83Ig5HxKqIWN/6f/UukuueisAPDv5la1KhuzzX+jbgnwFXSrqn9Th7PKc7uNYY/jXArcAPgJsjYqek6yVd1NrsM8BKSbuA99A7w6u2cl7rR0h+rX6h9d+ysyGcGDmvd6q5vIOZWQO5529m1kAO/mZmDeTgb2bWQA7+ZmYN5OBvZtZADv5mZg3k4G9m1kD/H9ySwkIjUk0UAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45842f7f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# A chainer link is a callable object. calling it performs the\n",
    "#   forward computation. (in this case, it performs Wx + b)\n",
    "model_y = model(data_x[:, None]) # chainer's link usually assumes input is [Batch Size, Input Dimension]\n",
    "# `model_y` is a chainer variable so we use `.data` to access its numpy array for plotting\n",
    "\n",
    "# we can plot the model's current fit in red. it should be terrible because we haven't trained it yet\n",
    "_ = plt.plot(data_x, model_y.data[:,0], c='r')\n",
    "_ = plt.scatter(data_x, data_y, c='b')\n",
    "_ = plt.title(\"Initial model\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss variable(0.330779492855072)\n",
      "dloss/dW [[-0.2403052]]\n"
     ]
    }
   ],
   "source": [
    "# now let's walk through how to perform forward computation\n",
    "#  and use AD to get gradients\n",
    "\n",
    "# first we clear the gradients that are stored in the model\n",
    "model.cleargrads()\n",
    "# as we have seen we can perform forward computation by calling the link\n",
    "model_y = model(data_x[:, None])\n",
    "\n",
    "# remember that `model_y` is a chainer variable. to operate on chainer variable\n",
    "#    we will use functions from chainer.functions to operate on those objects.\n",
    "loss = F.mean(F.square(model_y - data_y[:, None]))\n",
    "# `loss` is a scalar chainer variable\n",
    "assert isinstance(loss, chainer.Variable)\n",
    "print(\"loss\", loss)\n",
    "# calculating gradients d loss /d params is as simple as\n",
    "loss.backward()\n",
    "\n",
    "# we can inspect the gradient of loss with respect to W\n",
    "print(\"dloss/dW\", model.W.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to calculate gradients, we can code up a simple loop to perform gradient descent to train this model:\n",
    "\n",
    "(Hint: if you run into weird problems, maybe the state has been messed up and you can try re-runing all the code blocks from the beginning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Itr 0 loss: variable(0.32387468218803406)\n",
      "Itr 25 loss: variable(0.32387468218803406)\n",
      "Itr 50 loss: variable(0.32387468218803406)\n",
      "Itr 75 loss: variable(0.32387468218803406)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEICAYAAABfz4NwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAFj9JREFUeJzt3XGU3WV95/H3pwkBjbWojBaTAClm10bYRXsNeFbdXQQbigpu3T2wHEk91shKDt1tu0s8i4qc9Q85Xdx1TVVUrLaNcWu1plrLqtV64lbMRGMhIJsEsUnAGioYwSpEvvvHfcbzY5zJ3JnMZJiZ9+uce+Y+z+/5Pfd58su5n/k9v9+dm6pCkqSfm+0BSJIeHwwESRJgIEiSGgNBkgQYCJKkxkCQJAEGgnREST6TZN1sj0M6FuLnEPR4lORu4Der6nOzPRZpofAMQQtWksWzPYajNR/moMcPA0FzTpKXJdmZ5IEk/zfJP+ts25hkb5IfJLk9ySs7234jyZeTvCPJPwDXtrptSX4vyf1JvpXkgs4+X0zym539j9R2ZZIvtdf+XJJNSf7oCPO4qM3jUBvz2lZ/d5LzOu2uHeknyWlJKslrk/wd8FdtWWvDqL6/keTftOfPTvLZJN9LcmeSfzf1f33NZwaC5pQkzwVuAl4PPA14L7A1yfGtyV7gRcAvAG8F/ijJyZ0uzgbuAp4BvK1TdydwEnA98IEkGWcIR2q7GfhqG9e1wKuPMI81wIeB/wycCLwYuHui+Xf8S+CXgV8FPgJc2ul7NXAq8OkkS4HPtrE9HbgE+P3WRnoMA0FzzXrgvVV1S1X9pKo+BPwYOAegqv6kqu6pqker6qPAbmBNZ/97qup/VdXhqvrHVvftqnpfVf0E+BBwMv3AGMuYbZOcAjwfeHNVPVxV24CtR5jHa4GbquqzbawHquqbk/h3uLaqHmpz+ARwVpJT27bLgI9X1Y+BlwF3V9UH25y/Dvwp8G8n8VpaIAwEzTWnAr/TloseSPIAsAJ4JkCSyzvLSQ8AZ9D/bX7EvjH6/M7Ik6r6YXv6pHFef7y2zwS+16kb77VGrKB/NjNVP+27qn4AfJr+b//QP1v44/b8VODsUf9elwG/eBSvrXnKC1Kaa/YBb6uqt43e0H5Dfh/wEuBvquonSXYC3eWfmbqt7l7gqUme2AmFFUdovw84fZxtDwFP7JTHevMePY+PAG9J8iXgBOALndf566o6/0iDl8AzBD2+HZfkhM5jMf03/CuSnJ2+pUkuTPLzwFL6b5QHAZK8hv4Zwoyrqm8Dw/QvVC9J8gLg5UfY5QPAa5K8JMnPJVmW5Nlt207gkiTHJekBrxpgCH9B/2zgOuCjVfVoq/8U8E+SvLr1d1yS5yf55anMU/ObgaDHs78A/rHzuLaqhoHXAe8C7gf2AL8BUFW3A/8d+Bvg74EzgS8fw/FeBrwA+AfgvwEfpX9942dU1VeB1wDvAL4P/DX9N3SAN9E/e7if/oXxzRO9cLte8HHgvG77tpz0UvrLSffQX/J6O3D8GN1ogfODadIMSfJR4JtV9ZbZHos0CM8QpGnSlmJOb0tAa4GLgD+b7XFJg/KisjR9fpH+ss3TgP3Af2i3eUpzgktGkiTAJSNJUjOnloxOOumkOu2002Z7GJI0p+zYseO+qhqaqN2cCoTTTjuN4eHh2R6GJM0pSb49SDuXjCRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpGagQEiytn0X654kG8fYfkWSW9sXk2wb/fV8SU5J8mCS3x20T0nSsTVhICRZBGwCLgBWA5eO8X2sm6vqzKo6i/73zN4wavsNwGcm2ack6Rga5AxhDbCnqu6qqoeBLfT/iuNPVdWhTnHkS0oASHIx8C1g12T6lCQdW4MEwjIe+92w+1vdYyS5Msle+mcIV7W6JwFX0/+Sj0n32fpYn2Q4yfDBgwcHGK4kaSqm7aJyVW2qqtPpB8A1rfpa4B1V9eBR9HtjVfWqqjc0NOGf4pAkTdEgf8voAI/9svDlrW48W4B3t+dnA69Kcj1wIvBokh8BOybZpyRphg0SCNuBVUlW0n/TvgT4990GSVZV1e5WvBDYDVBVL+q0uRZ4sKre1b4s/Yh9SpKOrQkDoaoOJ9kA3AwsAm6qql1JrgOGq2orsCHJecAj9L8YfN1U+jzKuUiSjsKc+sa0Xq9X/vlrSZqcJDuqqjdROz+pLEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkYMBASLI2yZ1J9iTZOMb2K5LcmmRnkm1JVrf6Na1uZ5JvJHllZ5+7O/sMT9+UJElTsXiiBkkWAZuA84H9wPYkW6vq9k6zzVX1ntb+FcANwFrgNqBXVYeTnAx8I8mfV9Xhtt+/rqr7pnE+kqQpGuQMYQ2wp6ruqqqHgS3ARd0GVXWoU1wKVKv/YefN/4SReknS488ggbAM2Ncp7291j5HkyiR7geuBqzr1ZyfZBdwKXNEJiAL+T5IdSdaP9+JJ1icZTjJ88ODBAYYrSZqKabuoXFWbqup04Grgmk79LVX1HOD5wBuTnNA2vbCqngdcAFyZ5MXj9HtjVfWqqjc0NDRdw5UkjTJIIBwAVnTKy1vdeLYAF4+urKo7gAeBM1r5QPv5XeAT9JemJEmzZJBA2A6sSrIyyRLgEmBrt0GSVZ3ihcDuVr8yyeL2/FTg2cDdSZYm+flWvxR4Kf0L0JKkWTLhXUbtDqENwM3AIuCmqtqV5DpguKq2AhuSnAc8AtwPrGu7vxDYmOQR4FHgDVV1X5JfAj6RZGQMm6vqL6d7cpKkwaVq7tz40+v1anjYjyxI0mQk2VFVvYna+UllSRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqRkoEJKsTXJnkj1JNo6x/YoktybZmWRbktWtfk2r25nkG0leOWifkqRja8JASLII2ARcAKwGLh15w+/YXFVnVtVZwPXADa3+NqDX6tcC702yeMA+JUnH0CBnCGuAPVV1V1U9DGwBLuo2qKpDneJSoFr9D6vqcKs/YaR+kD4lScfWIIGwDNjXKe9vdY+R5Moke+mfIVzVqT87yS7gVuCKFhAD9dn2X59kOMnwwYMHBxiuJGkqpu2iclVtqqrTgauBazr1t1TVc4DnA29McsIk+72xqnpV1RsaGpqu4UqSRhkkEA4AKzrl5a1uPFuAi0dXVtUdwIPAGVPoU5I0wwYJhO3AqiQrkywBLgG2dhskWdUpXgjsbvUrkyxuz08Fng3cPUifkqRja/FEDarqcJINwM3AIuCmqtqV5DpguKq2AhuSnAc8AtwPrGu7vxDYmOQR4FHgDVV1H8BYfU7z3CRJk5CqmrjV40Sv16vh4eHZHoYkzSlJdlRVb6J2flJZkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkhoDQZIEGAiSpMZAkCQBBoIkqTEQJEnAgIGQZG2SO5PsSbJxjO1XJLk1yc4k25KsbvXnJ9nRtu1Icm5nny+2Pne2x9Onb1qSpMlaPFGDJIuATcD5wH5ge5KtVXV7p9nmqnpPa/8K4AZgLXAf8PKquifJGcDNwLLOfpdV1fD0TEWSdDQGOUNYA+ypqruq6mFgC3BRt0FVHeoUlwLV6r9eVfe0+l3AE5Icf/TDliRNt0ECYRmwr1Pez2N/ywcgyZVJ9gLXA1eN0c+vA1+rqh936j7YlovelCRjvXiS9UmGkwwfPHhwgOFKkqZi2i4qV9WmqjoduBq4prstyXOAtwOv71RfVlVnAi9qj1eP0++NVdWrqt7Q0NB0DVeSNMoggXAAWNEpL29149kCXDxSSLIc+ARweVXtHamvqgPt5w+AzfSXpiRJs2SQQNgOrEqyMskS4BJga7dBklWd4oXA7lZ/IvBpYGNVfbnTfnGSk9rz44CXAbcdzUQkSUdnwruMqupwkg307xBaBNxUVbuSXAcMV9VWYEOS84BHgPuBdW33DcCzgDcneXOreynwEHBzC4NFwOeA903jvCRJk5Sqmu0xDKzX69XwsHepStJkJNlRVb2J2k14hjAfvPXPd3H7PYcmbihJj0Orn/lk3vLy58z46/inKyRJwAI5QzgWySpJc51nCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCRgwEJKsTXJnkj1JNo6x/YoktybZmWRbktWt/vwkO9q2HUnO7ezzK61+T5J3Jsn0TUuSNFkTBkKSRcAm4AJgNXDpyBt+x+aqOrOqzgKuB25o9fcBL6+qM4F1wB929nk38DpgVXusPZqJSJKOziBnCGuAPVV1V1U9DGwBLuo2qKpDneJSoFr916vqnla/C3hCkuOTnAw8uaq+UlUFfBi4+CjnIkk6CosHaLMM2Ncp7wfOHt0oyZXAbwNLgHNHbwd+HfhaVf04ybLWT7fPZWO9eJL1wHqAU045ZYDhSpKmYtouKlfVpqo6HbgauKa7LclzgLcDr59CvzdWVa+qekNDQ9MzWEnSzxgkEA4AKzrl5a1uPFvoLP8kWQ58Ari8qvZ2+lw+iT4lSTNskEDYDqxKsjLJEuASYGu3QZJVneKFwO5WfyLwaWBjVX15pEFV3QscSnJOu7vocuCTRzUTSdJRmTAQquowsAG4GbgD+N9VtSvJdUle0ZptSLIryU761xHWjdQDzwLe3G5J3Znk6W3bG4D3A3uAvcBnpm1WkqRJS/8mn7mh1+vV8PDwbA9DkuaUJDuqqjdROz+pLEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSYyBIkgADQZLUGAiSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkYMBASLI2yZ1J9iTZOMb2K5LcmmRnkm1JVrf6pyX5QpIHk7xr1D5fbH3ubI+nT8+UJElTsXiiBkkWAZuA84H9wPYkW6vq9k6zzVX1ntb+FcANwFrgR8CbgDPaY7TLqmr46KYgSZoOg5whrAH2VNVdVfUwsAW4qNugqg51ikuBavUPVdU2+sEgSXocGyQQlgH7OuX9re4xklyZZC9wPXDVgK//wbZc9KYkGatBkvVJhpMMHzx4cMBuJUmTNW0XlatqU1WdDlwNXDPALpdV1ZnAi9rj1eP0e2NV9aqqNzQ0NF3DlSSNMkggHABWdMrLW914tgAXT9RpVR1oP38AbKa/NCVJmiWDBMJ2YFWSlUmWAJcAW7sNkqzqFC8Edh+pwySLk5zUnh8HvAy4bTIDlyRNrwnvMqqqw0k2ADcDi4CbqmpXkuuA4araCmxIch7wCHA/sG5k/yR3A08GliS5GHgp8G3g5hYGi4DPAe+b1plJkiYlVTXbYxhYr9er4WHvUpWkyUiyo6p6E7Xzk8qSJMBAkCQ1BoIkCTAQJEmNgSBJAgwESVJjIEiSAANBktQYCJIkwECQJDUGgiQJMBAkSY2BIEkCDARJUmMgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJjYEgSQIMBElSM1AgJFmb5M4ke5JsHGP7FUluTbIzybYkq1v905J8IcmDSd41ap9fafvsSfLOJJmeKUmSpmLCQEiyCNgEXACsBi4decPv2FxVZ1bVWcD1wA2t/kfAm4DfHaPrdwOvA1a1x9opzUCSNC0GOUNYA+ypqruq6mFgC3BRt0FVHeoUlwLV6h+qqm30g+GnkpwMPLmqvlJVBXwYuHjq05AkHa3FA7RZBuzrlPcDZ49ulORK4LeBJcC5A/S5f1Sfy8ZqmGQ9sB7glFNOGWC4kqSpmLaLylW1qapOB64GrpnGfm+sql5V9YaGhqarW0nSKIMEwgFgRae8vNWNZwsTL/8caP0M2qckaYYNEgjbgVVJViZZAlwCbO02SLKqU7wQ2H2kDqvqXuBQknPa3UWXA5+c1MglSdNqwmsIVXU4yQbgZmARcFNV7UpyHTBcVVuBDUnOAx4B7gfWjeyf5G7gycCSJBcDL62q24E3AH8APAH4THtIkmZJ+jf5zA29Xq+Gh4dnexiSNKck2VFVvYna+UllSRJgIEiSGgNBkgQYCJKkxkCQJAEGgiSpMRAkSYCBIElqDARJEmAgSJIaA0GSBBgIkqTGQJAkAQaCJKkxECRJgIEgSWoMBEkSYCBIkpo59RWaSQ4C357i7icB903jcOaChThnWJjzXohzhoU576nM+dSqGpqo0ZwKhKORZHiQ7xSdTxbinGFhznshzhkW5rxncs4uGUmSAANBktQspEC4cbYHMAsW4pxhYc57Ic4ZFua8Z2zOC+YagiTpyBbSGYIk6QgMBEkSsAACIcnaJHcm2ZNk42yPZ6YkWZHkC0luT7IryW+1+qcm+WyS3e3nU2Z7rNMtyaIkX0/yqVZemeSWdsw/mmTJbI9xuiU5McnHknwzyR1JXjDfj3WS/9T+b9+W5CNJTpiPxzrJTUm+m+S2Tt2YxzZ972zz/9skzzua157XgZBkEbAJuABYDVyaZPXsjmrGHAZ+p6pWA+cAV7a5bgQ+X1WrgM+38nzzW8AdnfLbgXdU1bOA+4HXzsqoZtb/BP6yqp4N/HP685+3xzrJMuAqoFdVZwCLgEuYn8f6D4C1o+rGO7YXAKvaYz3w7qN54XkdCMAaYE9V3VVVDwNbgItmeUwzoqruraqvtec/oP8GsYz+fD/Umn0IuHh2RjgzkiwHLgTe38oBzgU+1prMxzn/AvBi4AMAVfVwVT3APD/WwGLgCUkWA08E7mUeHuuq+hLwvVHV4x3bi4APV99XgBOTnDzV157vgbAM2Ncp729181qS04DnArcAz6iqe9um7wDPmKVhzZT/AfwX4NFWfhrwQFUdbuX5eMxXAgeBD7alsvcnWco8PtZVdQD4PeDv6AfB94EdzP9jPWK8Yzut73HzPRAWnCRPAv4U+I9Vdai7rfr3GM+b+4yTvAz4blXtmO2xHGOLgecB766q5wIPMWp5aB4e66fQ/214JfBMYCk/u6yyIMzksZ3vgXAAWNEpL29181KS4+iHwR9X1cdb9d+PnEK2n9+drfHNgH8BvCLJ3fSXA8+lv7Z+YltWgPl5zPcD+6vqllb+GP2AmM/H+jzgW1V1sKoeAT5O//jP92M9YrxjO63vcfM9ELYDq9qdCEvoX4TaOstjmhFt7fwDwB1VdUNn01ZgXXu+DvjksR7bTKmqN1bV8qo6jf6x/auqugz4AvCq1mxezRmgqr4D7EvyT1vVS4DbmcfHmv5S0TlJntj+r4/MeV4f647xju1W4PJ2t9E5wPc7S0uTV1Xz+gH8GvD/gL3Af53t8czgPF9I/zTyb4Gd7fFr9NfUPw/sBj4HPHW2xzpD8/9XwKfa818CvgrsAf4EOH62xzcD8z0LGG7H+8+Ap8z3Yw28FfgmcBvwh8Dx8/FYAx+hf53kEfpng68d79gCoX8n5V7gVvp3YU35tf3TFZIkYP4vGUmSBmQgSJIAA0GS1BgIkiTAQJAkNQaCJAkwECRJzf8HotXVBttchuUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45a18a3278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X/0HXV95/HnK4kBviSgJBEpId9QpVSqe6xkaT1uK63YInsK3bUVNCB0tala9nhWa0vltMdi02pt3eqqram2KolSSrc2rbhUqa7bLrh82VoquGBAQsLPL4lgUn6EJO/9Y+aayc2de+d+Z+7cuXdej3Pu+d47d+7MZ27gPXPfn/fnM4oIzMysXRaNuwFmZlY/B38zsxZy8DczayEHfzOzFnLwNzNrIQd/M7MWcvC3sZK0WNJeSWtGsO03SfpK1dstuO+dks4usN4LJOXWW0s6Q9I/S9oj6a2SPi7pXZU21lppybgbYJNF0t7MyxngaeBA+vqXImLLMNuLiAPAsoqaN41+Dfi7iHhn9xuSzgE+HhFra2+VTTwHfxtKRHwvUEu6F3hTRHwpb31JSyJifx1tm1KzwJfH3QibPk77WKUk/bakP5f0WUl7gIslvUzSzZIek/SgpA9Jela6/hJJIWlt+npz+v4X0lTHTZJOzWz/DElfkrRb0v+T9JrMe6sk/a2k70q6GTiVHJ10i6TL0hTNbkm/KOlHJP1L2tYPZtZfJOk3JW2X9IikT0o6LvP+Zel7j0q6omtfiyS9S9Ld6fvXSHpOge/yq8CPAX+cpsa+P/1+3i3peOBvgDXpe3slPTf9/j+brrdH0jckvTSzzdWS/krSvKRvS/rlzHs/Kun/pt/fw5Leny6fkfQZSbvS7+X/SFo5qP3WbA7+Ngr/AfgMcDzw58B+4G3ASuDlwLnAL/X5/OuB3wBOAO4D3gMgaRnwReDTwHOB9cAmSaenn/sjYA/wPGAD8J8KtHUd8HzgYuBDwBXATwIvIjlxvTxd703pOmen6z8H+GDarhcDH07bfTLwfWkbOv4L8O+BHwdWA3vTffUVET8O3AS8OSKWRcQ9mfceB34GuC99b1lEPJK+/bPA1cCzgS909iVpEfC3wC1pO18FvFPSK9PP/Tfg/RFxHPAC4Lp0+S+QpPhWAyuAtwJPDWq/NZuDv43CP0TE30TEwYh4MiJuiYivRcT+NIBtAl7R5/PXRcRcRDwDbAFeki6/ALgrIj6dbutW4HPAz6W/JH4W+I2IeCIibiMJgIO8JyKejojrgX3A5oiYj4idwD8AP5yutx74/Yj4dkTsAd4FvD4NqD8PfC4i/jEink7fU2YfbwbeFRH3R8RTwG8BP59+dhT+Z0TckPanXM2h7+9lwHER8TsRsS8itgGfAC5K338GOE3SiojYExFfyyxfCbwgIg6k/zbZvh+bQA7+Ngo7si8k/aCkz0t6SNJ3gatIgkmehzLPn+BQh/As8PI09fCYpMeAC4GTgBOBxV373j6ooRHxcOblk0D3686+v69re9uBpcCq9L3v7TcNjLsz664B/ibT5n9Jlz93UPsWqPv7OzZ9PkuSJsp+f7/KoV8pvwCcAdyZpnbOS5d/EvgScK2k+yW9V5L7Cyecg7+NQnfp4seAb5BcOR4H/CaHXxkXtQO4MSKenXksi4jLSYL2QeCUzPpVlo8+QBI8s9veB8wDD2b3m6anTsisuxN4VVe7j46IbJBeiGGn5N0BfKurHcsj4mcAIuLOiLiI5KT0B8BfSjo6/ZXw7oh4IfDvSNJ660u23cbMwd/qsBx4HPhXSS+kf76/n63AD0l6vaRnpY+zJJ2epog+B/yWpGMkvQi4pJrmA/BZ4O2S1kpaDmwEPhsRB4G/AC5IO7aPAn6bwwPzHwO/o3QsQ9oxe34FbXoYWJm2p4ibgH2S3iHpaCVjLF4s6cy0XZdIWpke0+PpMRyU9JOSXpSmqb5LkgY6WEH7bYwc/K0O7wAuJemM/RhJJ/DQ0k7OnybpeH2QJL3xu8BR6SpvIemIfZgkl/1npVp9uD8haff/Au4hOZa3pe26LX1+LXB/2q7sVf0HgP8B3KikAup/A/+2bIMi4hvAXwL3pmmcvmmktOT2POAs4F7gUZJ/j07V0nnAN9M2/j5wYUTsI0lr/XeSwH87SQroM2Xbb+Ml38zFzKx9fOVvZtZCDv5mZi3k4G9m1kIO/mZmLdTYgRorV66MtWvXjrsZZmYT5dZbb300IlYNWq+xwX/t2rXMzc2NuxlmZhNF0sCR7eC0j5lZKzn4m5m1kIO/mVkLOfibmbWQg7+ZWQs5+JuZFbRlC6xdC4sWJX+3bBl3ixausaWeZmZNsmULbNgATzyRvN6+PXkNsH4C727gK38zswKuvPJQ4O944olkeRnj+jXhK38zswLuu2+45UWM89eEr/zNzApYk3NT0LzlRYzq10QRDv5m1ihN7VTduBFmZg5fNjOTLF+oUfyaKMrB38wao5MG2b4dIg6lQZpwAli/HjZtgtlZkJK/mzaVS8+M4tdEUY29jeO6devCE7uZtcvatUnA7zY7C/feW3drRq875w/Jr4kyJxVJt0bEukHr+crfzBpjnGmQcRjFr4miHPzNrDHGmQapyrB9FuvXJ79qDh5M/tY1ZsDB38waYxSdqnVqcp9Ft0qCv6Q/lfSIpG/kvC9JH5K0TdJtkl5axX7NbLqMMw1ShXGWbg6rqiv/TwLn9nn/1cBp6WMD8EcV7dfMpsy40iBVKNpn0YRy1kqCf0R8FdjdZ5ULgE9H4mbg2ZJOqmLfZmbj1gnmecWTa9YcWkeCSy4Zf2qorpz/ycCOzOud6TIzs4mWzfP3MjMD5513+DrdJ4lxpIYa1eEraYOkOUlz8/Pz426OmdlAvfL8HZ0+i+uvz1+no+5y1rqC//3AKZnXq9Nlh4mITRGxLiLWrVq1qqammZktXF7Qlg71WRQJ7HWXs9YV/LcCb0irfn4UeDwiHqxp32ZmI1NkbMKgwD6OctaqSj0/C9wEnC5pp6Q3SnqzpDenq1wP3ANsA/4EeGsV+zUzG7ciYxN6rSMlf8dVzlrJfP4R8boB7wfwy1Xsy8ysSTpB+8ork/TOmjVJsM8G8yLr1M03czEzW4AtW4YL5uvXN2vMQqOqfczMmmDQIKxJmsYhj4O/mVlGkcA+SdM45HHwN7PW6nWFXySwT8PU0875m1kr5d08PW8wVjawr1nTe0TvJE097St/M5t62Sv8lSuTx8UX977CX7y49zaygX3Sp54GB38zm3LdOfxdu5JHngMHBgf2SZ96Ghz8zWzK9Zt7p5dOIB8U2Cd56mlw8DezKTdMJ2znCr+qwN6EefvzOPib2VQr2glbdeqm6WMBHPzNbKr16pzNmpmBzZurT900fSyAg7+ZVaKpKY7uztkVK5JH0Y7ahR5X08cCuM7fzErLq5mHZnSELnRenTLH1fSxAL7yN7PSmpLiqPrXR5njavpYAAd/MyutCSmOUXSwljmupo8FUOTdbn7M1q1bF3Nzc+NuhpkVsHZt7xTH7GzSkTqpbWjCcQ1L0q0RsW7Qer7yN7PSmpDiGMWvjyYc16g4+JtZaaNKcQyTwy9yL91hNT11U4bTPmbWSN2VNpBcdecF32HXn1a1pn0knSvpTknbJF3R4/01kr4s6Z8k3SbpvCr2a2b1G0U9/zDz6l98ce/9TvNV+khERKkHsBi4G/h+YCnwz8AZXetsAt6SPj8DuHfQds8888wws2bZvDliZiYiqadJHjMzyfKqt5l93etRdr+92jE7GyElf6vcdp2AuSgQu6u48j8L2BYR90TEPuAa4ILucwxwXPr8eOCBCvZrZjUbRT1/3jbz5tWvar9ZTZ+HZxSqCP4nAzsyr3emy7LeDVwsaSdwPfCfe21I0gZJc5Lm5ufnK2iamVVpFBU1eZ/tNa9+lfvNasogtTrVVe3zOuCTEbEaOA+4WtIR+46ITRGxLiLWrVq1qqammVlRo6ioyftsdl79YT87rCYMUqtbFcH/fuCUzOvV6bKsNwLXAkTETcDRwMoK9m1mNRpF3Xu/bXbm1d+8OX+dKjqgR3FSa7oqgv8twGmSTpW0FLgI2Nq1zn3AKwEkvZAk+DuvYzZhRlFRU2SbeetANbn6aR7MlatIr/CgB0kq5y6Sqp8r02VXAefHoQqffySpBPo68FODtulqH7P6TVrFy+xs70qg2dnhtzVpx56HgtU+HuRlZsBkDpJatCgJ992k5BaMbeS5fcxsKJNY8dLGXH1VHPzNDJjMipdW5uor4uBvZsBkXkV7SoeFc/A3M2Byr6I75aAHD1Z/E/Zp5uBvZsBkXUU39Wbxk8TB38y+p86r6IUG8DbOwzMKDv5mDdC2K9kyAbxoVVLbvtNhuc7fbMwmsb6+rDL3xi1S29/G77TDdf5mE2IS6+sHGXTVXaastEhV0jR+p1Vz8Dcbs0msr++nSEqnTFlpkaqkaftOR8HB32zMqq6vryrXvdDtFLnqLlNWWqQqaRLHLNSuyARA43h4YjdriypvjVjVtspsR+o92Zp05D5GNZHaKG43OSkoOLHb2IN83sPB39qkqkBY1SyXedvpbKtf+6qcabOMaZmlc1hFg7+rfcymSFWzXOZtp6Nf5UybK22awNU+Zi1UVa570Pr9KmcmaaRwmzn4m02Rqubn6bWdbv0qZzzfTvM5+JtNkaquurPbybOQyhmPum0O5/zNrK+qcvjuC6iHc/5mVomqfk141G2zVBL8JZ0r6U5J2yRdkbPOayXdIel2SZ+pYr9mVo8qcvgeddssS8puQNJi4CPAq4CdwC2StkbEHZl1TgN+HXh5RHxH0nPL7tfMJsuaNb0nc/Oo2/Go4sr/LGBbRNwTEfuAa4ALutb5ReAjEfEdgIh4pIL9mtkE6HTybt+epI2yJuFOYdOqiuB/MrAj83pnuizrB4AfkPSPkm6WdG6vDUnaIGlO0tz8/HwFTTObbJNeHZOd5A2SgWOdE4Dr/8erdNpniP2cBpwNrAa+KunFEfFYdqWI2ARsgqTap6a2mTVSd3VMZ3ZMmJyA2auTN6LYvP02WlVc+d8PnJJ5vTpdlrUT2BoRz0TEt4G7SE4GZpaj6uqYcfyKcCdvc1UR/G8BTpN0qqSlwEXA1q51Pkdy1Y+klSRpoHsq2LfZ1KoycI7rvreeWrm5Sgf/iNgPXA7cAHwTuDYibpd0laTz09VuAHZJugP4MvDOiNhVdt9m02yYwDnoqn5UNfaD9lvVdBM2AkWm/hzHw1M6W9sVnZO+yHr95thf6NTHw7SvjVMrjwuez9+seYYNhN3rv+UtR36+3/z5/d6HiBUrFn7Tk6bM22+HKxr8PbePWU3Kzm3T6/PS4Hn3u9M92feOOQZ29UjAFqnGqereAVYtz+1j1jBl8+55ZZN5Fi/OD/ydGvvdu3u/X6RT2Z25k83B36wmRat38jpRh6nymZmBAwd6vycdmp+nTAB3Z+5kc/A3q0mRQNuvJLPoFXXnqj5vLv7sdsoEcN+xa8IV6RgYx8MdvjZtilTHDOq87f58v87WfvvLdiSvWJE8XI0zHSjY4esrf7OaFLlS7pca6r671qBJ0vL2B4f/uti1C558Eq6+2rdcbBNX+5g1SGf2y269qm+2bEk6ge+7L0nlbNxYLHAPsw+bPK72MZtAw+TgF3qDFc+3Y+Dgb9YodXSiukTTwMHfrHGquGViPy7RNHDwN2sdl2ga1HczFzNrkPXrHezbzlf+ZgVM+u0Uzbo5+Fvr5QX2znIJLrlkuBuh+GRhTec6f2u1vJk2L70UPvWp/InRIL8uvuzsnWZlFK3zd/C3Vssb8LR4cf7EaB15Uxd7EJWNkwd5WaM0NQ2SN7BpUOCH/Lp4D6KySeDgbyM3rpuHF5EXwBcv7v+5XnXxnRNc3o9pD6KyJqkk+Es6V9KdkrZJuqLPeq+RFJIG/iSx6TGqm4dXIW/A04YNRy7vTKTWqy4+e4LrxYOorGlKB39Ji4GPAK8GzgBeJ+mMHustB94GfK3sPm2yNDkNkjfg6aMfPXL51VcnV/W9Rt32OsF1eBCVNVHpDl9JLwPeHRE/nb7+dYCI+N2u9f4Q+CLwTuBXIqJvb647fKdHGzpAfT9ba4o6O3xPBnZkXu9Ml2Ub81LglIj4fL8NSdogaU7S3Pz8fAVNsyaY5rlknOe3STXyDl9Ji4APAO8YtG5EbIqIdRGxbtWqVaNumtVkWueScZ7fJlkVc/vcD5ySeb06XdaxHHgR8BUlPWbPA7ZKOn9Q6semxzTOJTMoz1/05ipm41DFlf8twGmSTpW0FLgI2Np5MyIej4iVEbE2ItYCNwMO/Dbx8jqspWJTMTd17IO1Q+ngHxH7gcuBG4BvAtdGxO2SrpJ0ftntmzVVmZuiNHnsg7WDp3cwW6Ayc/i0oQLKxsPTO5iNWJmO7CaPfbB2cPA3yxg2D7/QWy76Pro2bg7+Zqk68/DTPPbBJoODv1mqzjmIpnXsg00OB3+z1Kjy8HmppIWmjMyq4OBvlhpFHt4lndZUDv5Wu+yV8MqVyaMJA51GkYdv8nTW1m4O/lar7ivhXbuSRx1XxYMqeYrm4YepCHJJpzWVB3lZrfIGN2WNYqBTVTdVH3Y7HsxldfMgL2ukIle8C7kqHnQ1XlX6ZdjtuKTTmsrB32pVpPN0zZrhUitFOlWrSr8Mux2XdFpTOfhbrXpdCWfNzMB55w1XIVPkajzvpBMxXEfzQiqCXNJpTeTgb4VVMQVx95XwihXJI3tVfP31w6VW+l2Nd9q8ffuhG7B3G6aj2WkcmxoR0cjHmWeeGVa9zZsjZmcjpOTv5s3FPzczE5FcKyePmZninx+mbdl9ZB9S78/lfWbFiiPbLOVvf3Z2uHYO+x2a1QGYiwIx1tU+LdLkKYh7ta3ovnp9Vsq/r24e32zdpoGrfewIZSpeRl2v3u+WiNA/tZJNJcHCAj94Rk1rFwf/CVfXgKNRT0Hcrw1FKmQ6naqzs4MD/4oVztubOfhPsGHnjSkTwEfd0ZnXhk6qpxP4B53sBp3IZmbggx90+aXZ2Dt28x7u8B0sr6Mzr+OybKftKDs6i7StyDr9OozdOWttQMEO30oCNXAucCewDbiix/tvB+4AbgNuBGYHbdPBf7C8ypW8qpiIZleqDGpbkZNdHVVJZk1WNPiXrvaRtBi4C3gVsBO4BXhdRNyRWecngK9FxBOS3gKcHREX9tuuq30Ga9u8MYsW9c7nd1fpbNmSdCDfd1+STtq40Skda486q33OArZFxD0RsQ+4Brggu0JEfDkiOrUcNwOrK9hv67VtwFHRPguPqDUbrIrgfzKwI/N6Z7oszxuBL/R6Q9IGSXOS5ubn5yto2nRr27wxbTvZmY3Skjp3JuliYB3wil7vR8QmYBMkaZ8amzax1q+f3mDfrXOcTumYlVdF8L8fOCXzenW67DCSzgGuBF4REU9XsF9roTad7MxGqYq0zy3AaZJOlbQUuAjYml1B0g8DHwPOj4hHKtinmZmVUDr4R8R+4HLgBuCbwLURcbukqySdn672fmAZ8BeSvi5pa87mzMysBpWM8I2I6yPiByLi+RGxMV32mxGxNX1+TkScGBEvSR/n99+idatiOuVJ1eZjNxuVWjt8bWG6Z63sTOMA05//bvOxm42Sp3SeAG0bzJXV5mM3WwhP6TxFRj2dclmjTMsUuUuX00Fmw3PwnwCjnk4ZFh5Ih51ZdFh5x3jCCaPdr9m0c/CfAKMe2VomgJe5QUwRecfe2c+o9ms27Rz8J8Cop3Fo8h2+8o599+7R7tds2jn4T4hRTlZWJq+el5aJqC4P3+vY60iFmU0zB/8JVFVHZ2c7eQVfRfLqvdIyHdu3wyWXJFfsVXfIepI3s5KKTPo/jodv5tJbVTcr6bWd7m2uWDH45imdbfW7g9aobqrS5BvTmI0Ldd3MZVRc599bVXXvedvpbGvjxuSqvcjNUzrybrZSpp1mNhzX+U+pqjpY89aXFp5XL5Jvd4esWTM4+E+Yqjo6i2xn2Lx6v/z/oP2aWb0c/CdMVR2dRbYzbIlpdn1IPlO2nWY2GtOd83/sMfjAB2D5cli2LPmb93zZMlgyGfPcVXWD8jLbKfJZ30jdrH5Fc/7THfzvugtOP734+kcffeQJoddJouiyo4468vJ3CnTPtAnJVf003z/YbFI4+HccPJhEqb17Yc+eQ38HPe/1Xufv/v3F9r1kyeCTxPLl/PPdy/jrv1/Oju8s46iVy3nNZcv5iZ/p8ZmZmUacTDzTpllzOfiPSgQ8/fShE8OwJ46u5we+u5fFzxS8pfGiRcnJoOwvks6yY4+FxYuH/grySjrzSkDNrD5Fg/9kJLmbRErSQ0cfDatWld7c89fC/dufYTl7WMZelrGX5ez53uuVS/ew4fV7OeuFfX6t7Nhx+PLuiXr6mZkp1h+S+fuLK5dzz/wy9rCcvSxLW7ucE05ZDjyr9HdiZqNXyZW/pHOBDwKLgY9HxHu73j8K+DRwJrALuDAi7u23zcZe+VdsJAOjDhxITgAFf30c9jxv/aL/nSxdurBfJN3PO6+PProRqS6zSVHblb+kxcBHgFcBO4FbJG2NiDsyq70R+E5EvEDSRcD7gAvL7nsarFmTP9K2Y+iBUYsXHwqgC/C9Kp0H0iqdjwTr/+OTh50c/u4v9/CZTXt58pE9zJ6wh9eet5d1p/c5qTzwwOHLn3lmuGMp8Iuk0LJjj03OuGYtV0Xa5yxgW0TcAyDpGuACIBv8LwDenT6/DviwJEVTOxxqtHHjkZUz3eocGNXznrm/JNAM69fPwIknAvBTL4Gfek+JHe3bd+SvjUG/TLLLHn308M88+WTxfWf7TRbyy6R72YSUCJtlVfFf7cnAjszrncCP5K0TEfslPQ6sAB6tYP8TrVMaeeWVSaCVDs+w1D0wqt/c/pWWcS5dCitWJI8q7N8P//qvvdNYRTrmH3zwyBNNUUcffWTKqkyp8FFHVfOdmPXRqEsWSRuADQBrWjQPwPr1hwLruAdGNf1+wbmWLIHjj08eVeiUCC+0omv37uRsnl1+4ECxfT/rWeV+kXS/15ASYWuWKoL//cApmder02W91tkpaQlwPEnH72EiYhOwCZIO3wraNnGyJ4JxyOuDaNG5OJEtqz3ppPLbi4CnnipVFsxDDx3+et++4Y9l2FRXr/WWLXO/yRSoIvjfApwm6VSSIH8R8PqudbYClwI3AT8H/P005/vHffVeRq8+CM/JUwEJjjkmeVRQIgwkwb9IX0neL5cdOw5/b5gS4WOPrW4k/PLlya8dq1Xp4J/m8C8HbiAp9fzTiLhd0lUkNxXYCnwCuFrSNmA3yQliKvXsMN2QPK/zBLDQE1C2D2IST16tsnRpcru1E06oZnsHDvTuNyk6Ev6RR+Duuw//BVN01N9RR1XbCe8S4YE8wrdiTZj6wHPvWCNEJFVYCx1b0ut50alVFi8eLpU1aNmxx07MycTTO4xJE6Y+aMIJyKxynalVhu0r6fdr5amniu1bSk4AVUyr0uk3GVGJsKd3GJMmdJhObMWOWT/ZqVVWrqxmm/v3H3nyGOYk8uCDyezBCy0Rzv4S+cEfhGuuqea4CnDwr1gTOkybcAIymwhLlsCzn508qnDw4JH9JkV+pezdC8cdV00bCnLwr1iVHaYL7bRtwgnIrJUWLSo1tUqdWl+su2VLkiNftCj5u2VL+W2uX5/k1g8ePHQz9IW0a8OG5Ao+4lDVUJH2DXv7RTNrn1YH/zIBdtT6TbNQRPYEtHFj8rkqT3BmNtlaXe3T5KqYqqqGXPZp1i5Fq31ac+XfK73T5KqYvM7ZYTtty/6CMLPp1Irgn5feyRsY2YSqmI0bkyv0rIV02jb5BGdm49OK4J939QvVBNhRqKrTtqpfEGY2XVoR/POucnfvbnZVTBVVQ1X9gjCz6dKK4N/v6reKAFvUKMpKB23fZZ9m1ksrBnk1YdDTqGf7HLR9B3szy2pNqee459gfdVlpk8tWzaw+LvXsUlV6Z6Gpm1FX3biqx8yG0ZrgX4UyI4JHXXXjqh4zG4aD/xDKDJjqVXUjJSeQKjp/XdVjZsNw8B9CmdRKtuoGksDf6W6pYk4hV/WY2TBa0+FbRqezuFeHKgzfqerOWTMblVo6fCWdIOmLkr6V/n1Oj3VeIukmSbdLuk3ShWX2Wbdsnr+XQamVSZtTyMzaoWza5wrgxog4Dbgxfd3tCeANEfFDwLnAH0qq6LY5o9crz98xKLUyiXMKmVk7lB3kdQFwdvr8U8BXgF/LrhARd2WePyDpEWAV8FjJfdci72pcGpyiyesgPuaY5BeD77RlZuNS9sr/xIh4MH3+EHBiv5UlnQUsBe7OeX+DpDlJc/Pz8yWbVo0yJZSTOqeQmU2/gcFf0pckfaPH44LsepH0HOf2Hks6Cbga+IWI6Hk7kojYFBHrImLdqlWrhjyU8nrl5/uVUA4a8NWUOYXMzLoNTPtExDl570l6WNJJEfFgGtwfyVnvOODzwJURcfOCWztCvebGueSSJFe/YkWSqtm9+9DUEDB4rp4mzClkZtZL2bTPVuDS9PmlwF93ryBpKfBXwKcj4rqS+xuZXvn5ThXsrl3w5JNw9dWHrtKLDPhy7b2ZNVWpOn9JK4BrgTXAduC1EbFb0jrgzRHxJkkXA38G3J756GUR8fV+2667zj/vnrlZ2Tr8qu6xa2ZWpaJ1/qWqfSJiF/DKHsvngDelzzcDm8vspw5r1uTX8ndkO3Dz1ne5pplNAk/vkOrVsdstG9g9l46ZTTIH/1SvuXeyugO78/lmNsk8t0+Ocd/8xcxsIWrJ+U8z3/rQzKaZ0z4VGfXN2c3MquQr/wqM+ubsZmZV85V/Bcrc4cvMbBymOvjXlYrx/PxmNmmmNviXudn6sHzzdDObNFMb/OtMxXjAl5lNmqkN/nWmYjzgy8wmzdQG/36pmFH0BXh+fjObJFMb/PNSMeedV19fgJlZU01t8M9LxVx/vcsyzcxaN7eP5+E3s2lWdG6fqb3yz+OyTDOzFgZ/l2WambUw+Lss08yspRO7ebpmM2u7Ulf+kk6Q9EVJ30r/PqfPusdJ2inpw2X2aWZm5ZVN+1wB3BgRpwE3pq/zvAcYE4NPAAAEB0lEQVT4asn9mZlZBcoG/wuAT6XPPwX8bK+VJJ0JnAj8Xcn9mZlZBcoG/xMj4sH0+UMkAf4wkhYBfwD8yqCNSdogaU7S3Pz8fMmmmZlZnoEdvpK+BDyvx1uHjYmNiJDUa8TYW4HrI2KnpL77iohNwCZIBnkNapuZmS1MqRG+ku4Ezo6IByWdBHwlIk7vWmcL8GPAQWAZsBT4aET06x9A0jywfcGNG7+VwKPjbkSNfLzTrU3HO+nHOhsRqwatVDb4vx/YFRHvlXQFcEJE/Gqf9S8D1kXE5Qve6YSQNFdkiPW08PFOtzYdb1uOtWzO/73AqyR9CzgnfY2kdZI+XrZxZmY2GqUGeUXELuCVPZbPAW/qsfyTwCfL7NPMzMpr3fQONdo07gbUzMc73dp0vK041sZO6WxmZqPjK38zsxZy8DczayEH/wq1aaK7Iscq6SWSbpJ0u6TbJF04jraWIelcSXdK2paWM3e/f5SkP0/f/5qktfW3shoFjvXtku5I/y1vlDQ7jnZWZdDxZtZ7jaSQNFXlnw7+1WrTRHdFjvUJ4A0R8UPAucAfSnp2jW0sRdJi4CPAq4EzgNdJOqNrtTcC34mIFwD/FXhfva2sRsFj/SeScTr/BrgO+L16W1mdgseLpOXA24Cv1dvC0XPwr1abJrobeKwRcVdEfCt9/gDwCDBw5GGDnAVsi4h7ImIfcA3JcWdlv4frgFdq0DwmzTTwWCPiyxHxRPryZmB1zW2sUpF/W0gu0t4HPFVn4+rg4F+tSie6a7iBx5ol6SySqT3uHnXDKnQysCPzeme6rOc6EbEfeBxYUUvrqlXkWLPeCHxhpC0arYHHK+mlwCkR8fk6G1aXVt7Jq4w6J7obtwqOtbOdk4CrgUsj4mC1rbS6SboYWAe8YtxtGZX0Iu0DwGVjbsrIOPgPKSLOyXtP0sOSTspMdPdIj9VeBvyYpLeSTnQnae+gie7GoYJjRdJxwOeBKyPi5hE1dVTuB07JvF6dLuu1zk5JS4DjgV31NK9SRY4VSeeQnPxfERFP19S2URh0vMuBFwFfSS/SngdslXR+OoPBxHPap1pbgUvT55cCf929QkSsj4g1EbGWJPXz6SYG/gIGHqukpcBfkRzjdTW2rSq3AKdJOjU9lotIjjsr+z38HPD3MZkjJwceq6QfBj4GnB8RPU/2E6Tv8UbE4xGxMiLWpv+v3kxy3FMR+MHBv2ptmuiuyLG+Fvhx4DJJX08fLxlPc4eX5vAvB24AvglcGxG3S7pK0vnpap8AVkjaBryd/hVejVXwWN9P8mv1L9J/y+4T4cQoeLxTzdM7mJm1kK/8zcxayMHfzKyFHPzNzFrIwd/MrIUc/M3MWsjB38yshRz8zcxa6P8DKMakVMcaCiQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45757f4ba8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now we can perform gradient descent to improve this model\n",
    "model = L.Linear(in_size=1, out_size=1)\n",
    "losses = []\n",
    "\n",
    "for i in range(100):\n",
    "    model.cleargrads()\n",
    "    loss = F.mean(F.square(model(data_x[:, None]) - data_y[:, None]))\n",
    "    losses.append(float(loss.data))\n",
    "    ## *** YOUR CODE HERE TO PERFORM GRADIENT DESCENT ***\n",
    "    ## Hint: you could access gradients with model.W.grad, model.b.grad\n",
    "    ## Hint2: you could write data into a parameter with model.W.data[:] = some_numpy_array\n",
    "    ## Hint3: if your model doesn't learn, remember to try different learning rates\n",
    "    if i % 25 == 0:\n",
    "        print(\"Itr\", i, \"loss:\", loss)\n",
    "plt.plot(np.array(losses))\n",
    "plt.title(\"Learning curve\")\n",
    "plt.figure()\n",
    "plt.plot(data_x, model(data_x[:, None])[:,0].data, c='r')\n",
    "plt.scatter(data_x, data_y, c='b')\n",
    "_ = plt.title(\"Trained model fitness\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train your first deep model\n",
    "\n",
    "Now we have learned the basics of Chainer. We can use it to train a deep model to classify MNIST digits. We will train a model on the MNIST dataset because the dataset is small.\n",
    "\n",
    "First we load the data and see what the images look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train[i][0] is the ith image that's flattened, and has shape: (784,)\n",
      "train[i][1] is the ith image's label, such as: 3\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAEQJJREFUeJzt3X2wXHV9x/H3Jw8kJDyYGAwZCCRAWkl1jHoljEHEUgEzQ8HBZsIUmioaZ4Rap+iUoaNQR6cMihZBKVEiARTrFGkiZBDMFFOkRi40kGDkOQHSPAABEqLk8ds/9oS5hLu/vdk9u2dzf5/XzJ2793z3nPO9m/vJ2T2/s/tTRGBm+RlSdQNmVg2H3yxTDr9Zphx+s0w5/GaZcvjNMuXwZ0zSvZI+3el1rTs4/IOApNWS/qLqPuqRNFvSY5JelbRR0gJJh1TdV+4cfuuEXwMzIuJQ4BhgGPC1alsyh38QkzRG0h2SXpD0cnH7yL3udqyk30raLGmhpLF91j9R0v2SXpH0sKRTmukjIp6LiBf7LNoFHNfMtqw8Dv/gNgT4IXA0cBTwR+Dave7zN8CngAnATuA7AJKOAO6kdoQeC3wRuE3SYXvvRNJRxX8QR9VrRNJJkl4FtgDnAP/a2q9mrXL4B7GIeCkibouIP0TEFuDrwIf3utvNEbEyIrYCXwZmSRoKnAcsjojFEbE7Iu4BeoGZ/ezn2Yh4W0Q8m+jlvuJp/5HAN4DVpfyS1jSHfxCTNErS9ZLWSNoMLAXeVoR7j+f63F4DDAfGUXu28FfFEf0VSa8AJ1F7htC0iFgL3AX8pJXtWOuGVd2AtdXFwJ8C0yNivaRpwP8C6nOfiX1uHwXsAF6k9p/CzRHxmTb0NQw4tg3btX3gI//gMVzSyD5fw4CDqb3Of6U4kXdZP+udJ2mqpFHAV4H/iIhdwC3AmZJOlzS02OYp/ZwwbEjSX+85HyDpaGovP5Y0+XtaSRz+wWMxtaDv+bqc2km1A6kdyX9D7en23m4GbgTWAyOBz0PtDD1wFnAp8AK1ZwJfop+/meKE32uJE35TgfslbaU27PcY0I5nFLYP5A/zMMuTj/xmmXL4zTLl8JtlyuE3y1RHx/kP0IgYyehO7tIsK6+zle2xTY3v2WL4JZ0BXA0MBX4QEVek7j+S0UzXqa3s0swSlsXAL59o+ml/cYnod4GPURvHPVfS1Ga3Z2ad1cpr/hOAJyPi6YjYTu1a7bPKacvM2q2V8B/Bm98U8nyx7E0kzZXUK6l3B9ta2J2ZlantZ/sjYl5E9EREz3BGtHt3ZjZArYR/LW9+R9iRxTIz2w+0Ev4HgCmSJks6AJgNLCqnLTNrt6aH+iJip6SLgF9QG+qbHxGPltaZmbVVS+P8EbGY2ltJzWw/48t7zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Yph98sUy3N0ttNho4Zk6w/d8Hxyfqw19Pbf2Xa9rq14QfVrwHcN+O6ZP1TT30iWX98/WHJejvt3Hhgsj554c5kfdiSB8tsx0rUUvglrQa2ALuAnRHRU0ZTZtZ+ZRz5PxIRL5awHTPrIL/mN8tUq+EP4G5JD0qa298dJM2V1CupdwfbWtydmZWl1af9J0XEWknvAO6R9PuIWNr3DhExD5gHcIjGRov7M7OStHTkj4i1xfeNwO3ACWU0ZWbt13T4JY2WdPCe28BpwMqyGjOz9mrlaf944HZJe7bz44i4q5SumrDqX6Yk60+eeW2HOulPeqx84ZQ706unf7VK7TxnV7L+nZffWbc2787Tkused/PLyfrulb9P1i2t6fBHxNPAe0rsxcw6yEN9Zply+M0y5fCbZcrhN8uUw2+WqUHzlt6vfeS2yva9fHv6ba1X/d/pHerkrZY9MylZnz55dbI+5aCNyfpXxq1I1v9hzBP1a+fVrwHMWPG5ZP1QX1XSEh/5zTLl8JtlyuE3y5TDb5Yph98sUw6/WaYcfrNMDZpx/ltmpd8ees27Dk3Wx6x8tel9D9nyx2R959Orm952q44j/bbYlxqs/8rbxyfrP//NmmT9zFGbG+yhvpdmpj9P/dBbmt604SO/WbYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5apQTPOv/vhVcn6oQ83WL+VfbewbrdbN7v+R28DnDnql01v++Xd6esjJs4f2vS2rTEf+c0y5fCbZcrhN8uUw2+WKYffLFMOv1mmHH6zTA2acX7r35CRI5P1J+anx/Hv/9A3GuwhPf14yuzz/y5ZH37vg01v2xpreOSXNF/SRkkr+ywbK+keSU8U38e0t00zK9tAnvbfCJyx17JLgCURMQVYUvxsZvuRhuGPiKXApr0WnwUsKG4vAM4uuS8za7NmX/OPj4h1xe31QN0PepM0F5gLMJJRTe7OzMrW8tn+iAggEvV5EdETET3DGdHq7sysJM2Gf4OkCQDF9/RUrmbWdZoN/yJgTnF7DrCwnHbMrFMavuaXdCtwCjBO0vPAZcAVwE8lXQCsAWa1s0lL23rO9Lq1l2b/IbnuYx+c32Dr6XH812Jbsj7j2ovr1iY+kP6QhcH8OQndoGH4I+LcOqVTS+7FzDrIl/eaZcrhN8uUw2+WKYffLFMOv1mm/Jbe/cCO03qS9buvvqZubYTa+0+8O+pe3AnAQc/VH7CLnTvLbsf2gY/8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmPM6/H3jmE0rW2z2Wn3LIkPRHg//6yu/VrV36xfcl171tyYnJ+jG3v56s69fLk/Xc+chvlimH3yxTDr9Zphx+s0w5/GaZcvjNMuXwm2VK0eD92GU6RGNjuvyhv/tq28wPJOujvrS2bu3ySekpFd5/wNCmeuoGO9mVrL/zzs/VrU39+vr0ttc811RPVVsWS9gcm9IXhhR85DfLlMNvlimH3yxTDr9Zphx+s0w5/GaZcvjNMuVx/kFu6PFTkvXthx+crG+dcECy/tJfpqcAf/RDP6xbG8KAhqPb4pPPnpKsb5ixNb2B3elrDKpS6ji/pPmSNkpa2WfZ5ZLWSlpefM1spWEz67yBPO2/ETijn+Xfjohpxdfictsys3ZrGP6IWAps6kAvZtZBrZzwu0jSI8XLgjH17iRprqReSb072NbC7sysTM2G/zrgWGAasA64qt4dI2JeRPRERM9wRjS5OzMrW1Phj4gNEbErInYD3wdOKLctM2u3psIvaUKfHz8OrKx3XzPrTg3H+SXdCpwCjAM2AJcVP08DAlgNfDYi1jXamcf587Pxog/Wrf35J3+TXPfKw3vLbmfAjl9wYbI++dL/6VAn+2ZfxvkbzvYQEef2s/iGfe7KzLqKL+81y5TDb5Yph98sUw6/WaYcfrNMeYpua6t3XHt/3dqj16ffLvzp//5wsv6Dib9qqqcBmZx+q/Jg4CO/WaYcfrNMOfxmmXL4zTLl8JtlyuE3y5TDb5Ypj/NbZWLH9mT93hXvSW+gjeP8empU27bdLXzkN8uUw2+WKYffLFMOv1mmHH6zTDn8Zply+M0y5XH+Dhh2zKRk/bELD0/WD308/UnM467vzo+RbkTD0n9+06c+1bZ9/zHS1xgcvqw7p+Auk4/8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmGo7zS5oI3ASMpzYl97yIuFrSWODfgUnUpumeFREvt6/V7jVs8tHJ+skLH03WF439WbJ+5rTTk/VuHpEeNumourXfXZK+vuHJSf9Wdjtv+O7L707WR/78t23bd7cYyJF/J3BxREwFTgQulDQVuARYEhFTgCXFz2a2n2gY/ohYFxEPFbe3AKuAI4CzgAXF3RYAZ7erSTMr3z695pc0CXgvsAwYHxHritJ6ai8LzGw/MeDwSzoIuA34QkRs7luLiKB2PqC/9eZK6pXUu4NtLTVrZuUZUPglDacW/B9FxJ6zUxskTSjqE4CN/a0bEfMioicieoYzooyezawEDcMvScANwKqI+Faf0iJgTnF7DrCw/PbMrF0G8pbeGcD5wApJy4tllwJXAD+VdAGwBpjVnha738Zr0s9ovjj2sZa2v2Pqkcn6sIder1vbvWVLS/secvDByfrj//xnyfrd53yzbm3SsNY+Hnuo0seuZ3a8Vrd255c/klz3QAb/UF/D8EfEfUC9N5SfWm47ZtYpvsLPLFMOv1mmHH6zTDn8Zply+M0y5fCbZcof3V2C15eOS9/hva1t/64f35Csf/XF+m9PfWrrYS3t+9jRLyTrd4z7XoMttG+q69Q4PsD5F19ctzb6P5eV3c5+x0d+s0w5/GaZcvjNMuXwm2XK4TfLlMNvlimH3yxTHucvwZGLNyXrHzjp3GT9gfff2tL+vzJuRf1ig0sQqtRomux33/H5ZH3S7buT9dG/8Fh+io/8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmPM5fgt0rf5+sj5+dfk/7B+ZcmKy/dvIfknU9VX/7J3/0keS6jfzq6eNaWv+gpfV7G7sqPX3bn9w7+D87v0o+8ptlyuE3y5TDb5Yph98sUw6/WaYcfrNMOfxmmVJEpO8gTQRuAsYDAcyLiKslXQ58Btjzwe6XRsTi1LYO0diYLs/qbdYuy2IJm2OTBnLfgVzksxO4OCIeknQw8KCke4ratyPim802ambVaRj+iFgHrCtub5G0Cjii3Y2ZWXvt02t+SZOoTT615/ORLpL0iKT5ksbUWWeupF5JvTtIX85pZp0z4PBLOgi4DfhCRGwGrgOOBaZRe2ZwVX/rRcS8iOiJiJ7hjCihZTMrw4DCL2k4teD/KCJ+BhARGyJiV0TsBr4PnNC+Ns2sbA3DL0nADcCqiPhWn+UT+tzt48DK8tszs3YZyNn+GcD5wApJy4tllwLnSppGbfhvNfDZtnRoZm0xkLP99wH9jRsmx/TNrLv5Cj+zTDn8Zply+M0y5fCbZcrhN8uUw2+WKYffLFMOv1mmHH6zTDn8Zply+M0y5fCbZcrhN8uUw2+WqYYf3V3qzqQXgDV9Fo0DXuxYA/umW3vr1r7AvTWrzN6OjojDBnLHjob/LTuXeiOip7IGErq1t27tC9xbs6rqzU/7zTLl8Jtlqurwz6t4/ynd2lu39gXurVmV9Fbpa34zq07VR34zq4jDb5apSsIv6QxJj0l6UtIlVfRQj6TVklZIWi6pt+Je5kvaKGlln2VjJd0j6Ynie79zJFbU2+WS1haP3XJJMyvqbaKk/5L0O0mPSvr7Ynmlj12ir0oet46/5pc0FHgc+CjwPPAAcG5E/K6jjdQhaTXQExGVXxAi6WTgNeCmiHhXsexKYFNEXFH8xzkmIv6xS3q7HHit6mnbi9mkJvSdVh44G/hbKnzsEn3NooLHrYoj/wnAkxHxdERsB34CnFVBH10vIpYCm/ZafBawoLi9gNofT8fV6a0rRMS6iHiouL0F2DOtfKWPXaKvSlQR/iOA5/r8/DwVPgD9COBuSQ9Kmlt1M/0YHxHritvrgfFVNtOPhtO2d9Je08p3zWPXzHT3ZfMJv7c6KSLeB3wMuLB4etuVovaarZvGagc0bXun9DOt/BuqfOyane6+bFWEfy0wsc/PRxbLukJErC2+bwRup/umHt+wZ4bk4vvGivt5QzdN297ftPJ0wWPXTdPdVxH+B4ApkiZLOgCYDSyqoI+3kDS6OBGDpNHAaXTf1OOLgDnF7TnAwgp7eZNumba93rTyVPzYdd109xHR8S9gJrUz/k8B/1RFD3X6OgZ4uPh6tOregFupPQ3cQe3cyAXA24ElwBPAL4GxXdTbzcAK4BFqQZtQUW8nUXtK/wiwvPiaWfVjl+irksfNl/eaZcon/Mwy5fCbZcrhN8uUw2+WKYffLFMOv1mmHH6zTP0/An4alFp2edUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45757036a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAEICAYAAACQ6CLfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAADuFJREFUeJzt3X+s1fV9x/HnC0RQKpUrjlGQ2iLtQpoUmxu1qW1xbp3abtgZiWxVmjmpqWYzcUuNy6LrtoQ0bZ3bOrfrdKLr7LpRA9lIV0rWmnab9WqpYukK2kuFXsQfdaBWhMt7f5wvzRXv+d7LOd9zvof7fj2Sk/s93/f3e75vTnjd74/POferiMDM8plSdwNmVg+H3ywph98sKYffLCmH3ywph98sKYc/MUnfkPS73V7XeoPDPwlIGpL0K3X30Yykv5X00qjHAUn76+4ruxPqbsAmv4i4Frj2yHNJ9wCHa2vIAO/5JzVJsyX9m6RnJf20mF5w1GKLJH1H0j5J6yX1jVr/PEn/JelFSd+TtKyCnmYClwFr230ta4/DP7lNAf4BeCuwEPgZ8NdHLXMV8DvAPOAQ8JcAkuYD/w78GdAH/AGwTtLpR29E0sLiF8TCCfR0GfAs8GAr/yCrjsM/iUXE8xGxLiJeiYj9wJ8DHzxqsfsiYmtEvAz8MbBC0lTgY8DGiNgYEYcjYhMwCFwyxnZ+HBGnRsSPJ9DWKuDe8JdKaufwT2KSTpb0d5J2StpHY297ahHuI54eNb0TmAbMoXG0cHmxR39R0ovA+TSOEFrtZyGwDLi31dew6viC3+R2I/BO4NyI2CNpKfBdQKOWOWPU9ELgIPAcjV8K90XENRX2cyXw7Yh4qsLXtBZ5zz95TJM0Y9TjBOAUGuf5LxYX8m4ZY72PSVoi6WTg08C/RsQI8I/Ar0v6NUlTi9dcNsYFw2NxFXBPG+tbhRz+yWMjjaAfedwK/AVwEo09+f8AXx1jvftoBHIPMAP4PYCIeBpYDtxM4wLd08AfMsb/meKC30tlF/wkvRdYAPxLK/84q5583cUsJ+/5zZJy+M2ScvjNknL4zZLq6jj/iZoeM5jZzU2apfIqL/NaHND4S7YZfkkXAbcDU4G/j4g1ZcvPYCbn6sJ2NmlmJR6KzRNetuXD/uIjol8ALgaWACslLWn19cysu9o55z8H2BERT0XEa8CXaHwoxMyOA+2Efz6v/1LIrmLe60haLWlQ0uBBDrSxOTOrUsev9kfEQET0R0T/NKZ3enNmNkHthH83r/9G2IJinpkdB9oJ/8PAYklvk3QicAWwoZq2zKzTWh7qi4hDkq4H/oPGUN/dEfFEZZ2ZWUe1Nc4fERtpfJXUzI4z/nivWVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUg6/WVIOv1lSDr9ZUm3dolvSELAfGAEORUR/FU2ZWee1Ff7CBRHxXAWvY2Zd5MN+s6TaDX8AX5P0iKTVYy0gabWkQUmDBznQ5ubMrCrtHvafHxG7Jf0CsEnSDyLiwdELRMQAMAAwS33R5vbMrCJt7fkjYnfxcy/wAHBOFU2ZWee1HH5JMyWdcmQa+BCwtarGzKyz2jnsnws8IOnI6/xTRHy1kq7MrONaDn9EPAW8u8JezKyLPNRnlpTDb5aUw2+WlMNvlpTDb5ZUFV/sseNZY6i2qamLziyt/+i355XWP/Dh7zatrTztodJ1P/ORy0rrI9u2l9atnPf8Zkk5/GZJOfxmSTn8Zkk5/GZJOfxmSTn8Zkl5nH8SmPqORU1rQ5fPLV33/cubj8MD/M38dS31NBHDI6+U1rW/vG7t8Z7fLCmH3ywph98sKYffLCmH3ywph98sKYffLCmP8/eAw+cvLa2/8Kny8e6vL72naW3WlBml6657eXZpffGma0rrOuFwaf2HF9zVtPZb264sXfekXT8qrVt7vOc3S8rhN0vK4TdLyuE3S8rhN0vK4TdLyuE3S8rj/BV45TfPLa3fsOb+0vr7T/p2af20KSeV1n/pm59sWnvL/SeWrjvzmz8orS/e90hp/fAHzy6tc0Hz0u5t5X9r4Cw8zt9J4+75Jd0taa+kraPm9UnaJGl78bP8kyJm1nMmcth/D3DRUfNuAjZHxGJgc/HczI4j44Y/Ih4EXjhq9nJgbTG9Fri04r7MrMNaPeefGxHDxfQeoOnJm6TVwGqAGZzc4ubMrGptX+2PiACipD4QEf0R0T+N6e1uzswq0mr4n5E0D6D4ube6lsysG1oN/wZgVTG9ClhfTTtm1i3jnvNLuh9YBsyRtAu4BVgDfFnS1cBOYEUnm+x1r8wp/x36V0O/XFr/9Cvl4/gnrj+1tP72td9pXjw8UrpuebWzpr6qGrdu44Y/IlY2KV1YcS9m1kX+eK9ZUg6/WVIOv1lSDr9ZUg6/WVL+Sm8F5gz8d/kCA+XlX6yula6b/id7Wl73rNueLK3XOQyZgff8Zkk5/GZJOfxmSTn8Zkk5/GZJOfxmSTn8Zkl5nN/acl6f/7z28cp7frOkHH6zpBx+s6QcfrOkHH6zpBx+s6QcfrOkPM5vHXXz3vc0rR1+/uhbQFo3ec9vlpTDb5aUw2+WlMNvlpTDb5aUw2+WlMNvlpTH+a3U1HcsKq1fN/u+0vrFj1/VtPbmQzta6smqMe6eX9LdkvZK2jpq3q2SdkvaUjwu6WybZla1iRz23wNcNMb82yJiafHYWG1bZtZp44Y/Ih4E/DlMs0mmnQt+10t6rDgtmN1sIUmrJQ1KGjzIgTY2Z2ZVajX8dwCLgKXAMPC5ZgtGxEBE9EdE/zSmt7g5M6taS+GPiGciYiQiDgN3AudU25aZdVpL4Zc0b9TTjwJbmy1rZr1p3HF+SfcDy4A5knYBtwDLJC0FAhgCPtHBHq1GQ5fPLa3PmjKjtD79jr4q27EKjRv+iFg5xuy7OtCLmXWRP95rlpTDb5aUw2+WlMNvlpTDb5aUv9JrpWac+3xp/RAjpfWZO37atFa+pnWa9/xmSTn8Zkk5/GZJOfxmSTn8Zkk5/GZJOfxmSXmc30q96/Th0vqa595dWh/Ztr3KdqxC3vObJeXwmyXl8Jsl5fCbJeXwmyXl8Jsl5fCbJeXwmyXl8Jsl5fCbJeXwmyXl8Jsl5fCbJeXwmyXl8JslNZFbdJ8B3AvMpXFL7oGIuF1SH/DPwJk0btO9IiKa/5F260lT55xWWv/sgg2l9U8OLR9nC88dY0fWLRPZ8x8CboyIJcB5wHWSlgA3AZsjYjGwuXhuZseJccMfEcMR8WgxvR/YBswHlgNri8XWApd2qkkzq94xnfNLOhM4G3gImBsRR/7G0x4apwVmdpyYcPglvQlYB9wQEftG1yIiaFwPGGu91ZIGJQ0e5EBbzZpZdSYUfknTaAT/ixHxlWL2M5LmFfV5wN6x1o2IgYjoj4j+aUyvomczq8C44Zck4C5gW0R8flRpA7CqmF4FrK++PTPrlIn86e73AVcCj0vaUsy7GVgDfFnS1cBOYEVnWrROGr7inaX106acVFp/+s7FpfVTPdTXs8YNf0R8C1CT8oXVtmNm3eJP+Jkl5fCbJeXwmyXl8Jsl5fCbJeXwmyXlW3Qn9+bf+Elb68/a+WpFnVi3ec9vlpTDb5aUw2+WlMNvlpTDb5aUw2+WlMNvlpTH+a3Uk4d+Vlqf9pP/K62PVNmMVcp7frOkHH6zpBx+s6QcfrOkHH6zpBx+s6QcfrOkPM6f3BULHi6tbznwltL6yPanqmzHush7frOkHH6zpBx+s6QcfrOkHH6zpBx+s6QcfrOkxh3nl3QGcC8wFwhgICJul3QrcA3wbLHozRGxsVONWmuG/vS9pfVrT72jtH7WNz5eWl/ElmNtyXrERD7kcwi4MSIelXQK8IikTUXttoj4bOfaM7NOGTf8ETEMDBfT+yVtA+Z3ujEz66xjOueXdCZwNvBQMet6SY9JulvS7CbrrJY0KGnwIAfaatbMqjPh8Et6E7AOuCEi9gF3AIuApTSODD431noRMRAR/RHRP43pFbRsZlWYUPglTaMR/C9GxFcAIuKZiBiJiMPAncA5nWvTzKo2bvglCbgL2BYRnx81f96oxT4KbK2+PTPrlIlc7X8fcCXwuKQj4zo3AyslLaUx/DcEfKIjHVpbDvYdbmv9uQ/4VG2ymsjV/m8BGqPkMX2z45g/4WeWlMNvlpTDb5aUw2+WlMNvlpTDb5aUIqJrG5ulvjhXF3Zte2bZPBSb2RcvjDU0/wbe85sl5fCbJeXwmyXl8Jsl5fCbJeXwmyXl8Jsl1dVxfknPAjtHzZoDPNe1Bo5Nr/bWq32Be2tVlb29NSJOn8iCXQ3/GzYuDUZEf20NlOjV3nq1L3BvraqrNx/2myXl8JslVXf4B2refple7a1X+wL31qpaeqv1nN/M6lP3nt/MauLwmyVVS/glXSTpfyXtkHRTHT00I2lI0uOStkgarLmXuyXtlbR11Lw+SZskbS9+jnmPxJp6u1XS7uK92yLpkpp6O0PSf0r6vqQnJP1+Mb/W966kr1ret66f80uaCvwQ+FVgF/AwsDIivt/VRpqQNAT0R0TtHwiR9AHgJeDeiHhXMe8zwAsRsab4xTk7Ij7VI73dCrxU923bi7tJzRt9W3ngUuDj1PjelfS1ghretzr2/OcAOyLiqYh4DfgSsLyGPnpeRDwIvHDU7OXA2mJ6LY3/PF3XpLeeEBHDEfFoMb0fOHJb+Vrfu5K+alFH+OcDT496vosa34AxBPA1SY9IWl13M2OYGxHDxfQeYG6dzYxh3Nu2d9NRt5XvmfeuldvdV80X/N7o/Ih4D3AxcF1xeNuTonHO1ktjtRO6bXu3jHFb+Z+r871r9Xb3Vasj/LuBM0Y9X1DM6wkRsbv4uRd4gN679fgzR+6QXPzcW3M/P9dLt20f67by9MB710u3u68j/A8DiyW9TdKJwBXAhhr6eANJM4sLMUiaCXyI3rv1+AZgVTG9ClhfYy+v0yu3bW92W3lqfu967nb3EdH1B3AJjSv+TwJ/VEcPTfp6O/C94vFE3b0B99M4DDxI49rI1cBpwGZgO/B1oK+HersPeBx4jEbQ5tXU2/k0DukfA7YUj0vqfu9K+qrlffPHe82S8gU/s6QcfrOkHH6zpBx+s6QcfrOkHH6zpBx+s6T+Hx9jbUzs60hCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f45756e3a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "train, test = chainer.datasets.get_mnist()\n",
    "# use train[data_point_index] to access data\n",
    "print(\"train[i][0] is the ith image that's flattened, and has shape:\", train[12][0].shape)\n",
    "print(\"train[i][1] is the ith image's label, such as:\", train[12][1])\n",
    "# here we visualize two of them\n",
    "plt.imshow(train[12][0].reshape([28, 28,]))\n",
    "plt.title(\"Label: %s\" % train[12][1])\n",
    "plt.figure()\n",
    "plt.imshow(train[42][0].reshape([28, 28,]))\n",
    "_ = plt.title(\"Label: %s\" % train[42][1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will provide some boilerplate code and train a linear classifier as an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss: 2.31815457344; accuracy: 0.125\n",
      "loss: 0.653798645735; accuracy: 0.82\n",
      "loss: 0.478858094811; accuracy: 0.8675\n",
      "loss: 0.371229023337; accuracy: 0.9\n",
      "loss: 0.344278301597; accuracy: 0.9025\n",
      "loss: 0.304191852808; accuracy: 0.9125\n",
      "loss: 0.324644305259; accuracy: 0.91\n",
      "loss: 0.359033640921; accuracy: 0.89\n",
      "loss: 0.299968173504; accuracy: 0.915\n",
      "loss: 0.307612841129; accuracy: 0.9325\n",
      "loss: 0.382457671463; accuracy: 0.9025\n",
      "loss: 0.27613604188; accuracy: 0.925\n",
      "loss: 0.368516498208; accuracy: 0.9025\n",
      "loss: 0.335841631591; accuracy: 0.92\n",
      "loss: 0.304305779338; accuracy: 0.9075\n",
      "loss: 0.302551475167; accuracy: 0.9125\n",
      "loss: 0.397146111727; accuracy: 0.8575\n",
      "loss: 0.305624783635; accuracy: 0.91\n",
      "loss: 0.3145725438; accuracy: 0.9075\n",
      "loss: 0.302962284982; accuracy: 0.91\n",
      "loss: 0.282926799506; accuracy: 0.9125\n",
      "loss: 0.333559719622; accuracy: 0.92\n",
      "loss: 0.237863760293; accuracy: 0.94\n",
      "loss: 0.360745558441; accuracy: 0.885\n",
      "loss: 0.337750664949; accuracy: 0.8925\n",
      "loss: 0.277246435136; accuracy: 0.93\n"
     ]
    }
   ],
   "source": [
    "def run(model, batchsize=16, num_epochs=2):\n",
    "    \n",
    "    optimizer = chainer.optimizers.Adam() # we will use chainer's Adam implementation instead of writing our own gradient based optimization\n",
    "    optimizer.setup(model)\n",
    "    \n",
    "    stats = defaultdict(lambda: deque(maxlen=25))\n",
    "    for epoch in range(num_epochs):\n",
    "        train_iter = chainer.iterators.SerialIterator(train, batchsize, repeat=False, shuffle=True)\n",
    "        test_iter = chainer.iterators.SerialIterator(test, batchsize, repeat=False, shuffle=False)\n",
    "        \n",
    "        for itr, batch in enumerate(train_iter):\n",
    "            xs = np.concatenate([datum[0][None, :] for datum in batch])\n",
    "            ys = np.array([datum[1] for datum in batch])\n",
    "\n",
    "            logits = model(xs)\n",
    "\n",
    "            loss = F.softmax_cross_entropy(logits, ys)\n",
    "            model.cleargrads()\n",
    "            loss.backward()\n",
    "            optimizer.update()\n",
    "            \n",
    "            # calculate stats\n",
    "            stats[\"loss\"].append(float(loss.data))\n",
    "            stats[\"accuracy\"].append(float((logits.data.argmax(1) == ys).sum() / batchsize))\n",
    "            if itr % 300 == 0:\n",
    "                print(\"; \".join(\"%s: %s\" % (k, np.mean(vs)) for k, vs in stats.items()))\n",
    "\n",
    "# try a simple linear model\n",
    "run(L.Linear(None, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we will try to improve performance by training an MLP instead. A partial implementation is provided for you to fill in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "NotImplementedError",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNotImplementedError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-6efe42bfbe2d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0;31m#     https://docs.chainer.org/en/stable/reference/functions.html#activation-functions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m         \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m \u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mMLP\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m200\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-9-06778ee8538a>\u001b[0m in \u001b[0;36mrun\u001b[0;34m(model, batchsize, num_epochs)\u001b[0m\n\u001b[1;32m     13\u001b[0m             \u001b[0mys\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mdatum\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mdatum\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mbatch\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m             \u001b[0mlogits\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mxs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m             \u001b[0mloss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msoftmax_cross_entropy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlogits\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mys\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-10-6efe42bfbe2d>\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, x)\u001b[0m\n\u001b[1;32m     14\u001b[0m         \u001b[0;31m# Hint: you should make use of non-linearities / activation functions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     15\u001b[0m         \u001b[0;31m#     https://docs.chainer.org/en/stable/reference/functions.html#activation-functions\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mNotImplementedError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m \u001b[0mrun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mMLP\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m200\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNotImplementedError\u001b[0m: "
     ]
    }
   ],
   "source": [
    "class MLP(chainer.Chain):\n",
    "\n",
    "    def __init__(self, n_units, n_out):\n",
    "        super(MLP, self).__init__()\n",
    "        with self.init_scope():\n",
    "            # the size of the inputs to each layer will be inferred\n",
    "            self.l1 = L.Linear(None, n_units)  # n_in -> n_units\n",
    "            self.l2 = L.Linear(None, n_units)  # n_units -> n_units\n",
    "            self.l3 = L.Linear(None, n_out)  # n_units -> n_out\n",
    "\n",
    "    def __call__(self, x):\n",
    "        # *** YOUR CODE HERE TO BUILD AN MLP W/ self.l1, self.l2, self.l3 ***\n",
    "        # \n",
    "        # Hint: you should make use of non-linearities / activation functions\n",
    "        #     https://docs.chainer.org/en/stable/reference/functions.html#activation-functions\n",
    "        raise NotImplementedError()\n",
    "run(MLP(200, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next you should try to implement logging test loss and see if the model is overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def better_run(model, batchsize=16, num_epochs=2):\n",
    "\n",
    "    optimizer = chainer.optimizers.Adam() # we will use chainer's Adam implementation instead of writing our own gradient based optimization\n",
    "    optimizer.setup(model)\n",
    "    \n",
    "    stats = defaultdict(lambda: deque(maxlen=25))\n",
    "    for epoch in range(num_epochs):\n",
    "        train_iter = chainer.iterators.SerialIterator(train, batchsize, repeat=False, shuffle=True)\n",
    "        for itr, batch in enumerate(train_iter):\n",
    "            xs = np.concatenate([datum[0][None, :] for datum in batch])\n",
    "            ys = np.array([datum[1] for datum in batch])\n",
    "\n",
    "            logits = model(xs)\n",
    "\n",
    "            loss = F.softmax_cross_entropy(logits, ys)\n",
    "            model.cleargrads()\n",
    "            loss.backward()\n",
    "            optimizer.update()\n",
    "            \n",
    "            # calculate stats\n",
    "            stats[\"loss\"].append(float(loss.data))\n",
    "            stats[\"accuracy\"].append(float((logits.data.argmax(1) == ys).sum() / batchsize))\n",
    "            if itr % 300 == 0:\n",
    "                test_iter = chainer.iterators.SerialIterator(test, batchsize, repeat=False, shuffle=False)\n",
    "                # *** YOUR CODE implement logging of stats on test set ***\n",
    "                print(\"; \".join(\"%s: %s\" % (k, np.mean(vs)) for k, vs in stats.items()))\n",
    "\n",
    "better_run(MLP(200, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try different variants!\n",
    "\n",
    "- Does using a ConvNet improve performance (reduce overfitting?)\n",
    "- Try changing the learning rate and observe the effect\n",
    "- Does the model train if you give it correlated gradients? (consecutively sample many batches of \"1\", then many batches of \"2\", ... etc"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}