reinforcement-learning-a-game-of-pong.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Reinforcement Learning - A Game of Pong"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Intro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The goal is to build a model to play Atari's Pong and learn the game only using the pixels of each frame as input in combination with negative and positive reinforcement from the environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First install software (for Mac OS):\n",
    "1. conda update conda\n",
    "2. conda update --all\n",
    "3. ```brew install cmake boost boost-python sdl2 swig wget```\n",
    "4. [Download OpenAI Gym](https://github.com/openai/gym/archive/master.zip)\n",
    "5. ```cd``` into the downloaded repo\n",
    "6. ```MACOSX_DEPLOYMENT_TARGET=10.11 python setup.py install```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## A Peek at Pong"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-05-10 11:23:28,294] Making new env: Pong-v0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dimensions of the frame are (210, 160, 3).\n",
      "Which if you were to flatten it into a vector would be 100800 pixels.\n",
      "This is what a frame looks like:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADhZJREFUeJzt3X+s1fV9x/Hna4BAFFP8UUaQDkiwCSzb7Upck1bj5lrR\nNKXuDwdZDG6mVxNn2qTLBppsZglJ50r9Y4s2EMnY4lA2auUPN4ekrWkyKmAoAooCQuQGLy1dpu3E\n9sJ7f3w/tz1e74Vzz/sczvecvR7Jzf2ez/f7Pef9zc2L7w++3/dRRGBmrfu1bhdg1uscIrMkh8gs\nySEyS3KIzJIcIrOkjoVI0nJJhyUdkbSmU59j1m3qxP8TSZoCvA58FjgJ7AZWRcShtn+YWZd1ak90\nA3AkIo5FxM+Bp4AVHfoss66a2qH3nQe81fD6JPC7Ey0s6YK7w4/OmdGmssyad3r47I8j4tqLLdep\nEF2UpEFgEGDWldO4+77ru1XKuL5089JJr7Pxuwc7UEnvO/v+85NeZ8b0WztQyeT8/SMHTzSzXKcO\n54aA+Q2vrytjvxQRGyJiWUQsmzlzSofKMOu8ToVoN7BY0kJJlwErge0d+iyzrurI4VxEjEj6M+B5\nYAqwKSJ8rGN9qWPnRBHxHPBcp97/UhvvfKeV8yYb/3ynlfOmuvAdC2ZJDpFZkkNkluQQmSU5RGZJ\nDpFZkkNkluQQmSU5RGZJDpFZkkNkluQQmSV17aG8XuObTdunl282HY/3RGZJDpFZkkNkluRzogm4\n6Uj71KHpSCe1vCeSNF/SdyQdknRQ0pfL+MOShiTtKz+3t69cs/rJ7IlGgK9GxMuSZgF7Je0o8x6N\niK/nyzOrv5ZDFBGngFNl+l1Jr1I1bZy0n46MsGv4TKulmHVVWy4sSFoAfAL4QRl6QNJ+SZskzW7H\nZ5jVVTpEkq4AtgFfiYh3gMeBRcAA1Z5q/QTrDUraI2nPyNnz2TLMuiYVIknTqAL0ZER8CyAihiPi\nXEScBzZSNbf/kMYOqFNn+Eq79a7M1TkBTwCvRsQ3GsbnNix2B3Cg9fLM6i9zde7TwF3AK5L2lbEH\ngVWSBoAAjgP3pio0q7nM1bnvAxpnVt90PTVrhk9GzJIcIrMkh8gsqRY3oF4xdSqfmnN1t8sw+4Dd\nvN3Uct4TmSU5RGZJDpFZkkNkluQQmSU5RGZJDpFZkkNkluQQmSU5RGZJDpFZUq1CtGv4jLv+WM+p\nVYjMelHqLm5Jx4F3gXPASEQsk3QV8DSwgOrx8Dsj4r9zZZrVVzv2RL8XEQMRsay8XgPsjIjFwM7y\n2qxvdeJ5ohXAzWV6M/Bd4C+bWdHPFFkvyu6JAnhB0l5Jg2VsTmkxDPA2MCf5GWa1lt0TfSYihiR9\nFNgh6bXGmRERkmK8FUvoBgFmXTktWYZZ96T2RBExVH6fBp6h6nY6PNrAsfw+PcG6v+yAOnPmlEwZ\nZl2V6YB6eflKFSRdDnyOqtvpdmB1WWw18Gy2SLM6yxzOzQGeqboJMxX4l4j4D0m7ga2S7gFOAHfm\nyzSrr0wH1GPAb48zfga4JVOUWS/xHQtmSQ6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSQ6R\nWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSS0/2Srp41SdTkctAv4K+AjwJeBHZfzBiHiu5QrN\nai7zePhhYABA0hRgiKrjz58Aj0bE19tSoVnNtetw7hbgaEScaNP7mfWMdoVoJbCl4fUDkvZL2iRp\ndps+w6yW0iGSdBnwBeBfy9DjVOdHA8ApYP0E6w1K2iNpz3vvncuWYdY17dgT3Qa8HBHDABExHBHn\nIuI8sJGqK+qHuAOq9Yt2hGgVDYdyoy2EizuouqKa9a3sl3xdDnwWuLdh+BFJA1TfGHF8zDyzvpMK\nUUT8DLh6zNhdqYrMeozvWDBLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySH\nyCzJITJLcojMkhwisySHyCwp9VCeWV2cff/5D7yeMf3WS/bZF90TlbZXpyUdaBi7StIOSW+U37Mb\n5q2VdETSYUmXbkvMuqSZw7l/BJaPGVsD7IyIxcDO8hpJS6h60C0t6zxWuqOa9a2LhigiXgR+MmZ4\nBbC5TG8Gvtgw/lREvB8RbwJHmKBlllm/aPXCwpyIOFWm3wbmlOl5wFsNy50sYx/i5o3WL9JX5yIi\nqNpjTXY9N2+0vtBqiIZHmzSW36fL+BAwv2G568qYWd9qNUTbgdVlejXwbMP4SknTJS0EFgMv5Uo0\nq7eL/j+RpC3AzcA1kk4Cfw18Ddgq6R7gBHAnQEQclLQVOASMAPdHhE94rK9dNEQRsWqCWbdMsPw6\nYF2mKLNe4tt+zJIcIrMkh8gsySEyS3KIzJIcIrMkP09kfeFSPj80lvdEZkkOkVmSQ2SW5BCZJTlE\nZkkOkVmSQ2SW5BCZJTlEZkkOkVlSqx1Q/07Sa5L2S3pG0kfK+AJJ70naV36+2cnizeqg1Q6oO4Df\njIjfAl4H1jbMOxoRA+XnvvaUaVZfLXVAjYj/jIiR8nIXVWsss/+X2nFO9KfAvze8XlgO5b4n6caJ\nVnIHVOsXqUchJD1E1RrryTJ0CvhYRJyR9Eng25KWRsQ7Y9eNiA3ABoA5vz5z0h1Uzeqi5T2RpLuB\nzwN/XFoJUxrZnynTe4GjwPVtqNOstloKkaTlwF8AX4iI/20Yv3b0q1QkLaLqgHqsHYWa1VWrHVDX\nAtOBHZIAdpUrcTcBfyPpF8B54L6IGPu1LGZ9pdUOqE9MsOw2YFu2KLNe4jsWzJIcIrMkh8gsySEy\nS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJIcIrMkh8gsySEyS3KIzJJa7YD6sKSh\nhk6ntzfMWyvpiKTDkrr3bbRml0irHVABHm3odPocgKQlwEpgaVnnsdHGJWb9qqUOqBewAniqtM56\nEzgC3JCoz6z2MudED5SG9pskzS5j84C3GpY5WcY+xB1QrV+0GqLHgUXAAFXX0/WTfYOI2BARyyJi\n2cyZPuKz3tVSiCJiOCLORcR5YCO/OmQbAuY3LHpdGTPrW612QJ3b8PIOYPTK3XZgpaTpkhZSdUB9\nKVeiWb212gH1ZkkDQADHgXsBIuKgpK3AIapG9/dHhE94rK+1tQNqWX4dsC5TlFkv8R0LZkkOkVmS\nQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJTlEZkkOkVmSQ2SW5BCZJbXavPHp\nhsaNxyXtK+MLJL3XMO+bnSzerA4u+mQrVfPGfwD+aXQgIv5odFrSeuB/GpY/GhED7SrQrO6aeTz8\nRUkLxpsnScCdwO+3tyyz3pE9J7oRGI6INxrGFpZDue9JujH5/ma118zh3IWsArY0vD4FfCwizkj6\nJPBtSUsj4p2xK0oaBAYBZl05LVmGWfe0vCeSNBX4Q+Dp0bHSg/tMmd4LHAWuH299d0C1fpE5nPsD\n4LWIODk6IOna0W+BkLSIqnnjsVyJZvXWzCXuLcB/AR+XdFLSPWXWSj54KAdwE7C/XPL+N+C+iGj2\nGyXMelKrzRuJiLvHGdsGbMuXZdY7fMeCWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSQ6RWZJD\nZJbkEJklOURmSQ6RWZJDZJaUfTy8LX46MsKu4TPdLsOsJd4TmSU5RGZJzTwePl/SdyQdknRQ0pfL\n+FWSdkh6o/ye3bDOWklHJB2WdGsnN8Cs25rZE40AX42IJcCngPslLQHWADsjYjGws7ymzFsJLAWW\nA4+NNi8x60cXDVFEnIqIl8v0u8CrwDxgBbC5LLYZ+GKZXgE8VdpnvQkcAW5od+FmdTGpc6LSTvgT\nwA+AORFxqsx6G5hTpucBbzWsdrKMmfWlpkMk6QqqTj5fGdvRNCICiMl8sKRBSXsk7Rk5e34yq5rV\nSlMhkjSNKkBPRsS3yvCwpLll/lzgdBkfAuY3rH5dGfuAxg6oU2f4IqH1rmauzgl4Ang1Ir7RMGs7\nsLpMrwaebRhfKWm6pIVUXVBfal/JZvXSzB0LnwbuAl4Z/TIv4EHga8DW0hH1BNVXrBARByVtBQ5R\nXdm7PyLOtb1ys5popgPq9wFNMPuWCdZZB6xL1GXWM3wyYpbkEJklOURmSQ6RWZJDZJak6maDLhch\n/Qj4GfDjbtfSRtfQP9vTT9sCzW/Pb0TEtRdbqBYhApC0JyKWdbuOdumn7emnbYH2b48P58ySHCKz\npDqFaEO3C2izftqeftoWaPP21OacyKxX1WlPZNaTuh4iSctLQ5MjktZ0u55WSDou6RVJ+yTtKWMT\nNnKpG0mbJJ2WdKBhrGcb0UywPQ9LGip/o32Sbm+Yl9ueiOjaDzAFOAosAi4Dfggs6WZNLW7HceCa\nMWOPAGvK9Brgb7td5wXqvwn4HeDAxeoHlpS/03RgYfn7Ten2NjSxPQ8Dfz7Osunt6fae6AbgSEQc\ni4ifA09RNTrpBxM1cqmdiHgR+MmY4Z5tRDPB9kwkvT3dDlG/NDUJ4AVJeyUNlrGJGrn0in5sRPOA\npP3lcG/08DS9Pd0OUb/4TEQMALdR9eW7qXFmVMcNPXsZtNfrLx6nOm0YAE4B69v1xt0OUVNNTeou\nIobK79PAM1SHAxM1cukVqUY0dRMRwxFxLiLOAxv51SFbenu6HaLdwGJJCyVdRtU5dXuXa5oUSZdL\nmjU6DXwOOMDEjVx6RV81ohn9B6G4g+pvBO3YnhpcSbkdeJ3qqshD3a6nhfoXUV3d+SFwcHQbgKup\n2iu/AbwAXNXtWi+wDVuoDnF+QXVOcM+F6gceKn+vw8Bt3a6/ye35Z+AVYH8Jztx2bY/vWDBL6vbh\nnFnPc4jMkhwisySHyCzJITJLcojMkhwisySHyCzp/wAaskqp1XfSOgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112515470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import gym\n",
    "import numpy as np\n",
    "import pickle\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "def display_frame(frame):\n",
    "    plt.imshow(frame, cmap='Greys', interpolation='nearest')\n",
    "    plt.show()\n",
    "    \n",
    "env = gym.make(\"Pong-v0\")\n",
    "frame = env.reset()\n",
    "print(\"The dimensions of the frame are {0}.\".format(frame.shape))\n",
    "print(\"Which if you were to flatten it into a vector would be {0} pixels.\".format(frame.size))\n",
    "print(\"This is what a frame looks like:\")\n",
    "display_frame(frame)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "100,800 pixels is a bit much for our purposes. We can reduce the size of each frame by:\n",
    "1. cropping it so that we only focus on pixels between the dark green boundaries\n",
    "2. reduce the image size by half by taking every other pixel\n",
    "3. remove color so that only a greyscale image remains\n",
    "4. remove the background\n",
    "5. set every pixel that's not black to white"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Process Frame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "crop\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQUAAAD8CAYAAAB+fLH0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADS1JREFUeJzt3X3MnfVdx/H3x5Zn6ICBtdK6lgUxuOhGCKJOYsIeAJFu\nMZklm2GOpDFBBJ0hMP7Y/lnCNp0uJrrgwBFFCDLImmUPMNxcTCyjdAVaykOBAa2lZV2U6Ris7Osf\n5+o8v653H87jfcb7lTQ953euu9f3XOfuh+s6PdyfVBWStMfPTHsASfOLoSCpYShIahgKkhqGgqSG\noSCpMbZQSHJ+kseSbElyzbj2I2m0Mo7PKSRZADwOvB3YCtwPXFJVj4x8Z5JGalxnCmcDW6rqqap6\nBbgNWDmmfUkaoYVj+nNPAZ7ru78V+LW5Nj7q6IW1aNFhYxpFEsDOHT/4TlWdfKDtxhUKB5RkNbAa\n4LhFh/H7l75xWqNIrwl/8/FNzxzMduO6fNgGLOu7v7Rb+7GquqGqzqqqs446asGYxpB0qMYVCvcD\npyVZkeRwYBWwZkz7kjRCY7l8qKrdSf4Y+AqwALipqjaNY1+SRmts7ylU1ReBL47rz5c0Hn6iUVLD\nUJDUMBQkNQwFSQ1DQVLDUJDUMBQkNQwFSY15Fwprd+xi7Y5d0x5Des2ad6EgaboMBUkNQ0FSw1CQ\n1JjaT16ayzmLXz/tEaTXNM8UJDUMBUkNQ0FSw1CQ1DAUJDUGDoUky5J8LckjSTYlubJbPzHJPUme\n6H4/YXTjShq3Yc4UdgMfrKozgHOAy5OcAVwD3FtVpwH3dvclzYiBQ6GqtlfV+u7294DN9OriVgI3\nd5vdDLxr2CElTc5I3lNIshx4C3AfsLiqtncPPQ8snuNrVidZl2TdSy+9OooxJI3A0KGQ5Fjgc8BV\nVfVi/2PV67nfZ9e9tXHS/DRUKCQ5jF4g3FJVd3bLO5Is6R5fAuwcbkRJkzTMvz4EuBHYXFWf7Hto\nDXBpd/tS4PODjydp0ob5H6J+E/gD4OEkG7q1DwHXA7cnuQx4BnjPcCNKmqSBQ6Gq/h3IHA+fN+if\nK2m6/ESjpIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpIahIKlh\nKEhqGAqSGoaCpIahIKkxih/xviDJt5J8obtvbZw0w0ZxpnAlvXaoPayNk2bYsL0PS4HfAT7Tt2xt\nnDTDhj1T+GvgauBHfWvWxkkzbJgymIuAnVX1wFzbWBsnzZ5hy2AuTnIhcCSwKMk/0dXGVdV2a+Ok\n2TNMFf21VbW0qpYDq4B/rar3YW2cNNPG8TmF64G3J3kCeFt3X9KMGOby4ceq6uvA17vbu7A2TppZ\nfqJRUsNQkNQwFCQ1DAVJDUNBUsNQkNQwFCQ1DAVJDUNBUsNQkNQwFCQ1DAVJDUNBUsNQkNQwFCQ1\nDAVJDUNBUsNQkNQYtgzm+CR3JHk0yeYkv25tnDTbhj1T+BTw5ar6JeBX6dXHWRsnzbBhymBeB5wL\n3AhQVa9U1X9hbZw004Y5U1gBvAD8Q9c6/Zkkx3CQtXGS5qdhfsT7QuBM4Iqqui/Jp9jrUqGqKsk+\na+OSrAZWAxy36LAhxpBm1w9e/spPrB15xDunMMn/G+ZMYSuwtaru6+7fQS8kdnR1ceyvNs4uSWl+\nGqY27nnguSSnd0vnAY9gbZw004ZtiLoCuCXJ4cBTwB/SC5rbk1wGPAO8Z8h9SJqgoUKhqjYAZ+3j\nIWvjpBnlJxolNQwFSQ1DQVLDUJDUMBQkNQwFSQ1DQVLDUJDUMBQkNQwFSQ1DQVLDUJDUMBQkNQwF\nSQ1DQVLDUJDUMBQkNYb9cWyShjDtn9y8L8PWxv1pkk1JNia5NcmR1sZJs22YhqhTgD8BzqqqNwEL\ngFVYGyfNtGHfU1gIHJVkIXA08J9YGyfNtGF6H7YBfwE8C2wH/ruq7sbaOGmmDXP5cAK9s4IVwM8D\nxyR5X/82VVXAnLVxSdYlWffSS68OOoakERvm8uFtwNNV9UJV/RC4E/gNrI2TZtowofAscE6So5OE\nXgHMZqyNk2bawJ9T6Jqm7wDWA7uBbwE3AMdibZw0s4atjfsw8OG9ll/G2jhpZvkxZ0kNQ0FSw1CQ\n1DAUJDUMBUkNQ0FSw1CQ1DAUJDUMBUkNQ0FSw1CQ1DAUJDUMBUkNQ0FSw1CQ1DAUJDUMBUkNQ0FS\n44ChkOSmJDuTbOxbm7MaLsm1SbYkeSzJ/CvKk7RfB3Om8Fng/L3W9lkNl+QMetVxv9x9zd8m8ee3\nSzPkgKFQVd8AvrvX8lzVcCuB26rq5ap6GtgCnD2iWSVNwKDvKcxVDXcK8Fzfdlu7NUkzYug3GvdX\nDbc/1sZJ89OgoTBXNdw2YFnfdku7tZ9gbZw0Pw0aCnNVw60BViU5IskK4DTgm8ONKGmSDtgQleRW\n4LeBk5JspdcIdT37qIarqk1JbgceoVcld3lVeW0gzZADhkJVXTLHQ/ushquqjwIfHWYoSdPjJxol\nNQwFSQ1DQVLDUJDUMBQkNQwFSQ1DQVLDUJDUMBQkNQwFSQ1DQVLDUJDUMBQkNQwFSQ1DQVLDUJDU\nMBQkNQwFSY1Ba+M+keTRJA8luSvJ8X2PWRsnzbBBa+PuAd5UVb8CPA5cC9bGST8NBqqNq6q7q2p3\nd3ctvX4HsDZOmnmjeE/hA8CXutvWxkkzbqhQSHIdvX6HWwb4WmvjpHlo4FBI8n7gIuC9XZ8kWBsn\nzbyBQiHJ+cDVwMVV9f2+h6yNk2bcoLVx1wJHAPckAVhbVX9kbZw0+watjbtxP9tbGyfNMD/RKKlh\nKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpIah\nIKlhKEhqGAqSGgN1SfY99sEkleSkvjW7JKUZNmiXJEmWAe8Anu1bs0tSmnEDdUl2/ope90P1rdkl\nKc24QctgVgLbqurBvR466C5Ja+Ok+emAvQ97S3I08CF6lw4Dq6obgBsAFv/cUXWAzSVNyCGHAvBG\nYAXwYNcOtRRYn+RsDqFLUtL8dMiXD1X1cFX9bFUtr6rl9C4Rzqyq57FLUpp5B/NPkrcC/wGcnmRr\nksvm2raqNgF7uiS/jF2S0swZtEuy//Hle923S1KaYX6iUVLDUJDUMBQkNQwFSQ1DQVLDUJDUMBQk\nNQwFSQ1DQVLDUJDUMBQkNQwFSQ1DQVLDUJDUMBQkNQwFSQ1DQVLDUJDUMBQkNQwFSY1Beh9G7n92\n72btjl3THkMSkKrplzMleQH4X+A7054FOAnn6OccrVme4w1VdfKBNpoXoQCQZF1VneUczuEc053D\n9xQkNQwFSY35FAo3THuAjnO0nKP1Uz/HvHlPQdL8MJ/OFCTNA1MPhSTnJ3ksyZYk10xwv8uSfC3J\nI0k2JbmyW/9Ikm1JNnS/LpzALN9O8nC3v3Xd2olJ7knyRPf7CWOe4fS+57whyYtJrprE8UhyU5Kd\nSTb2rc35/JNc232/PJbknWOe4xNJHk3yUJK7khzfrS9P8lLfcfn0mOeY83UY+fGoqqn9AhYATwKn\nAocDDwJnTGjfS4Azu9vHAY8DZwAfAf58wsfh28BJe619HLimu30N8LEJvy7PA2+YxPEAzgXOBDYe\n6Pl3r9GDwBHAiu77Z8EY53gHsLC7/bG+OZb3bzeB47HP12Ecx2PaZwpnA1uq6qmqegW4DVg5iR1X\n1faqWt/d/h6wGThlEvs+SCuBm7vbNwPvmuC+zwOerKpnJrGzqvoG8N29lud6/iuB26rq5ap6GthC\n7/toLHNU1d1Vtbu7uxZYOop9Heoc+zHy4zHtUDgFeK7v/lam8BczyXLgLcB93dIV3eniTeM+be8U\n8NUkDyRZ3a0trqrt3e3ngcUTmGOPVcCtffcnfTxg7uc/ze+ZDwBf6ru/ojuV/7ckvzWB/e/rdRj5\n8Zh2KExdkmOBzwFXVdWLwN/Ru5x5M7Ad+MsJjPHWqnozcAFweZJz+x+s3nniRP6ZKMnhwMXAv3RL\n0zgejUk+/7kkuQ7YDdzSLW0HfqF73f4M+Ocki8Y4wsReh2mHwjZgWd/9pd3aRCQ5jF4g3FJVdwJU\n1Y6qerWqfgT8PSM6Nd2fqtrW/b4TuKvb544kS7o5lwA7xz1H5wJgfVXt6Gaa+PHozPX8J/49k+T9\nwEXAe7uAojtd39XdfoDetfwvjmuG/bwOIz8e0w6F+4HTkqzo/gu1ClgziR0nCXAjsLmqPtm3vqRv\ns3cDG/f+2hHPcUyS4/bcpvfG1kZ6x+HSbrNLgc+Pc44+l9B36TDp49Fnrue/BliV5IgkK4DTgG+O\na4gk5wNXAxdX1ff71k9OsqC7fWo3x1NjnGOu12H0x2Mc754e4jutF9J75/9J4LoJ7vet9E5JHwI2\ndL8uBP4ReLhbXwMsGfMcp9J79/hBYNOeYwC8HrgXeAL4KnDiBI7JMcAu4HV9a2M/HvRCaDvwQ3rX\nxJft7/kD13XfL48BF4x5ji30rtn3fI98utv297rXawOwHvjdMc8x5+sw6uPhJxolNaZ9+SBpnjEU\nJDUMBUkNQ0FSw1CQ1DAUJDUMBUkNQ0FS4/8AMyHyw2TrhTgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b2b64e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reduce image size\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAC15JREFUeJzt3W+oJYV5x/Hvr+tat0k2/mm6XVxbJYhBCq7pkiqG0qrb\nmlRMXxmFFAkB36RFISVo3vVFwVcheVECi5oKsUmsUSISFGsMbSGxarRNdN2usYor6lrTYGpsWs3T\nF2eCN5tdd+695567c5/vB5Z7Zs5ZZgb53jnnODtPqgpJ/fzKeu+ApPVh/FJTxi81ZfxSU8YvNWX8\nUlPGLzW1qviTXJJkX5Knklw3r52StPay0ot8kmwC/h3YDRwAHgKurKon5rd7ktbKcav4ux8Anqqq\npwGSfAX4CHDE+Lf82nG1devmVWxS0tt59dX/4/WfvJExr11N/KcCzy1ZPgD83tv9ha1bN/PRq967\nik1KejtfveUHo1+75l/4Jbk6ycNJHn799TfXenOSRlpN/M8Dpy1Z3jGs+wVVtaeqdlXVri1bNq1i\nc5LmaTXxPwScmeSMJMcDVwB3zWe3JK21FX/mr6o3kvw5cC+wCbi5qh6f255JWlOr+cKPqvoG8I05\n7YukBfIKP6kp45eaMn6pKeOXmjJ+qSnjl5oyfqmpdYv/Oy+9wndeemW9Ni+155lfasr4paaMX2rK\n+KWmVvUPe1bjvG2nrNemJeGZX2rL+KWmjF9qyvilpoxfauqo8Se5OcnBJN9fsu7kJPcl2T/8PGlt\nd1PSvI058/8tcMkh664D7q+qM4H7h2VJE3LU+KvqH4EfHrL6I8Atw+NbgD+d835JWmMr/cy/rape\nGB6/CGyb0/5IWpBVf+FXszG/Rxz167gu6di00vhfSrIdYPh58EgvdFyXdGxaafx3AVcNj68Cvj6f\n3ZG0KGP+V9+XgW8DZyU5kOQTwA3A7iT7gYuHZUkTctR/1VdVVx7hqYvmvC+SFsgr/KSmjF9qyvil\npoxfasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2pqzD38\nTkvyQJInkjye5JphvSO7pAkbc+Z/A/hUVZ0NnAd8MsnZOLJLmrQx47peqKrvDo9/DOwFTsWRXdKk\nLeszf5LTgXOBB3FklzRpo+NP8k7ga8C1VfXq0ufebmSX47qkY9Oo+JNsZhb+rVV1x7B61Mgux3VJ\nx6Yx3/YHuAnYW1WfXfKUI7ukCTvqxB7gAuDPgO8leWxY9xlmI7puG8Z3PQtcvja7KGktjBnX9c9A\njvC0I7ukifIKP6kp45eaMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqkp45ea\nMn6pKeOXmjJ+qSnjl5oyfqmpMXfvPSHJvyT512FW318N653VJ03YmDP/T4ELq+ocYCdwSZLzcFaf\nNGljZvVVVf33sLh5+FM4q0+atLETezYN9+w/CNxXVaNn9TmuSzo2jRnaQVW9CexMciJwZ5LfOeT5\nSnLYWX1VtQfYA7DtN7cc9jXSRvc/P733l9ad8Kt/vA578pZlfdtfVT8CHgAuYeSsPknHpjHf9r9n\nOOOTZAuwG3gSZ/VJkzbmbf924JYkm5j9sritqu5O8m2c1SdN1phZff8GnHuY9a/grD5psrzCT2rK\n+KWmjF9qyvilpoxfasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2rK+KWmjF9qyvilpoxfamrU\n3Xslrc5636n3cEaf+Yd79z+a5O5h2XFd0oQt523/NcDeJcuO65ImbOzEnh3AnwA3LlntuC5pwsae\n+T8HfBr42ZJ1juuSJmzM0I5LgYNV9ciRXlNVxWx45+Ge21NVu6pq15Ytm1a+p5Lmasy3/RcAlyX5\nMHACsDXJlxjGdVXVC47rkqZnzIju66tqR1WdDlwBfLOqPobjuqRJW81FPjcAu5PsBy4eliVNxLIu\n8qmqbwHfGh47rkuaMC/vlZoyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnj\nl5oyfqkp45eaMn6pKeOXmjJ+qalRt/FK8gzwY+BN4I2q2pXkZOCrwOnAM8DlVfVfa7ObkuZtOWf+\nP6yqnVW1a1h2XJc0Yat52++4LmnCxsZfwD8keSTJ1cO6UeO6JB2bxt66+4NV9XyS3wDuS/Lk0ier\nqpIcdlzX8MviaoB3bd28qp2VND+jzvxV9fzw8yBwJ/ABhnFdAG83rstZfdKxacygznckedfPHwN/\nBHwfx3VJkzbmbf824M4kP3/931XVPUkeAm5L8gngWeDytdtNSfN21Pir6mngnMOsd1yXNGFe4Sc1\nZfxSU8YvNWX8UlPGLzVl/FJTxi81ZfxSU8YvNWX8UlPGLzVl/FJTxi81ZfxSU8YvNWX8UlPGLzVl\n/FJTo+JPcmKS25M8mWRvkvOTnJzkviT7h58nrfXOSpqfsWf+zwP3VNX7mN3Pby+O65Imbcytu98N\n/D5wE0BV/W9V/QjHdUmTNubMfwbwMvDFJI8muXG4f7/juqQJGxP/ccD7gS9U1bnAaxzyFr+qitk8\nv1+S5OokDyd5+PXX31zt/kqakzHxHwAOVNWDw/LtzH4ZOK5LmrCjxl9VLwLPJTlrWHUR8ASO65Im\nbeyU3r8Abk1yPPA08HFmvzgc1yVN1Kj4q+oxYNdhnnJclzRRXuEnNWX8UlPGLzVl/FJTxi81ZfxS\nU8YvNWX8UlPGLzVl/FJTxi81ZfxSU8YvNWX8UlPGLzVl/FJTxi81ZfxSU2OGdpyV5LElf15Ncq3j\nuqRpG3P33n1VtbOqdgK/C/wEuBPHdUmTtty3/RcBP6iqZ3FclzRpy43/CuDLw2PHdUkTNjr+4Z79\nlwF/f+hzjuuSpmc5Z/4PAd+tqpeGZcd1SRO2nPiv5K23/OC4LmnSRsU/jOTeDdyxZPUNwO4k+4GL\nh2VJEzF2XNdrwCmHrHsFx3VJk+UVflJTxi81ZfxSU8YvNWX8UlPGLzVl/FJTxi81ZfxSU8YvNWX8\nUlPGLzVl/FJTxi81ZfxSU8YvNWX8UlPGLzVl/FJTxi81ZfxSU5kN21nQxpKXgdeA/1zYRhfr19mY\nx+ZxTcdvV9V7xrxwofEDJHm4qnYtdKMLslGPzePamHzbLzVl/FJT6xH/nnXY5qJs1GPzuDaghX/m\nl3Rs8G2/1NRC409ySZJ9SZ5Kct0itz1PSU5L8kCSJ5I8nuSaYf3JSe5Lsn/4edJ67+tKJNmU5NEk\ndw/LG+W4Tkxye5Ink+xNcv5GObaVWFj8STYBfwN8CDgbuDLJ2Yva/py9AXyqqs4GzgM+ORzLdcD9\nVXUmcP+wPEXXAHuXLG+U4/o8cE9VvQ84h9kxbpRjW76qWsgf4Hzg3iXL1wPXL2r7a3xsXwd2A/uA\n7cO67cC+9d63FRzLDmYRXAjcPazbCMf1buA/GL7nWrJ+8se20j+LfNt/KvDckuUDw7pJS3I6cC7w\nILCtql4YnnoR2LZOu7UanwM+DfxsybqNcFxnAC8DXxw+0tyY5B1sjGNbEb/wW4Uk7wS+BlxbVa8u\nfa5mp5JJ/a+UJJcCB6vqkSO9ZorHNTgOeD/whao6l9ll5r/wFn/Cx7Yii4z/eeC0Jcs7hnWTlGQz\ns/Bvrao7htUvJdk+PL8dOLhe+7dCFwCXJXkG+ApwYZIvMf3jgtk7zQNV9eCwfDuzXwYb4dhWZJHx\nPwScmeSMJMcDVwB3LXD7c5MkwE3A3qr67JKn7gKuGh5fxey7gMmoquurakdVnc7sv883q+pjTPy4\nAKrqReC5JGcNqy4CnmADHNtKLfpf9X2Y2WfKTcDNVfXXC9v4HCX5IPBPwPd467PxZ5h97r8N+C3g\nWeDyqvrhuuzkKiX5A+Avq+rSJKewAY4ryU7gRuB44Gng48xOgJM/tpXwCj+pKb/wk5oyfqkp45ea\nMn6pKeOXmjJ+qSnjl5oyfqmp/wfTlCOn4WjzKAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115da17f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "remove color\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAC1VJREFUeJzt3U+sXOV5x/HvrzYoFUmDIS2yMNQgWUTeBEdWWxQWNJSK\npAiiLhBIkWgUxZu0MlKrxGTTdhGJVZQsqkiWQ2opbVKXNA3yIsh1iJpIEcX8aRNsXCiFYmTstE5L\nyI7ydDEHMXFt37n3ztzrmef7ka5mzjtz57yvrN+855x7/D6pKiT180vr3QFJ68PwS00Zfqkpwy81\nZfilpgy/1JThl5paVfiT3J7keJIXkuyZVqckzV5WepNPkg3AvwK3ASeAJ4B7q+ro9LonaVY2ruJ3\nfwN4oapeBEjyDeAu4LzhT+LthNKMVVUmed9qDvuvBl4Z2z4xtEmaA6uZ+SeSZBewa9b7kbQ8qwn/\nq8A1Y9tbhrZfUFV7gb3gYb90MVnNYf8TwLYk1yW5FLgHeGQ63ZI0ayue+avqzSR/CDwKbAAeqqpn\np9YzSTO14j/1rWhnHvZLM7cWV/slzTHDLzVl+KWmDL/UlOGXmjL8UlOGX2pq3cJfVVgzQFo/zvxS\nU4ZfasrwS00ZfqmpmS/mcT7JRP/3QNKMOPNLTRl+qSnDLzVl+KWmDL/U1JLhT/JQktNJfjzWdkWS\nQ0meHx43zbabkqZtkpn/L4Hbz2rbAxyuqm3A4WFb0hxZMvxV9Y/AmbOa7wL2D8/3Ax+bcr8kzdhK\nz/mvqqqTw/PXgKum1B9Ja2TVd/hVVV1oSW7LdUkXp5XO/KeSbAYYHk+f741VtbeqdlbVzhXuS9IM\nrDT8jwD3Dc/vA749ne5IWitLVuxJ8nXgFuB9wCngT4G/Bw4A1wIvA3dX1dkXBc/1WS7dI83YpBV7\nLNclLRjLdUm6IMMvNWX4paYMv9SU4ZeaMvxSU4ZfasrwS00Zfqkpwy81Zfilpgy/1JThl5oy/FJT\nhl9qyvBLTRl+qSnDLzU1Sbmua5I8luRokmeT7B7aLdklzbFJFvDcDGyuqqeSvAd4klGFnj8AzlTV\ng0n2AJuq6rNLfJZr+EkzNrU1/KrqZFU9NTz/GXAMuBpLdklzbVnn/Em2AjuAx7FklzTXJi7XleTd\nwDeB+6vq9eSdI4sLleyyXJd0cZpo3f4klwAHgUer6gtD23Hglqo6OVwX+F5V3bDE53jOL83Y1M75\nM5rivwIcezv4A0t2SXNskqv9NwPfB34EvDU0f47Ref+ySnY580uzZ7kuqSnLdUm6IMMvNWX4paYM\nv9SU4ZeaMvxSU4ZfasrwS00Zfqkpwy81Zfilpgy/1JThl5oy/FJThl9qyvBLTRl+qSnDLzVl+KWm\nJlm9911J/inJPw+1+v58aLdWnzTHJlm9N8BlVfXGsH7/D4DdwO9jrT7pojPNWn1VVW8Mm5cMP4W1\n+qS5NtE5f5INSZ4BTgOHqmriWn1JdiU5kuTIVHosaSqWtW5/ksuBbwF/BPygqi4fe+2nVXXB834P\n+9XV9u3b/1/b0aNHZ7KvmazbX1X/DTwG3A6cGmr0MTyeXm4nJa2fSa72/+ow45Pkl4HbgOewVp80\n1yYp0b0Z2J9kA6MviwNVdTDJD4EDST7JUKtvhv2UNGVLhr+q/gXYcY72/wJunUWnJM2ed/hJTRl+\nqSnDLzVl+KWmDL/UlOGXmjL8UlOGX2rK8EtNGX6pKcMvNWX4paYMv9SU4ZeaMvxSU4ZfasrwS00t\na/XeVe/M1XulmZv66r3D2v1PJzk4bFuuS5pjyzns3w0cG9veAxyuqm3A4WFb0pyYtGLPFuD3gH1j\nzZbrkubYpDP/F4HPAG+NtVmuS5pjkxTtuAM4XVVPnu89NbpqeM6LeVW1t6p2VtXOlXdT0rRNUrTj\nQ8CdST4KvAv4lSRfYyjXVVUnLdclzZ9JSnQ/UFVbqmorcA/w3ar6OJbrkubaam7yeRC4LcnzwO8M\n25LmhDf5SAtmJiW6JS0Owy81Zfilpgy/1JThl5oy/FJThl9qyvBLTRl+qSnDLzVl+KWmDL/UlOGX\nmjL8UlOGX2rK8EtNGX6pKcMvNTXJ6r0keQn4GfC/wJtVtTPJFcDfAFuBl4C7q+qns+mmpGlbzsz/\n21V149j6+5brkubYag77LdclzbFJw1/APyR5MsmuoW2icl2SLk4TnfMDN1fVq0l+DTiU5LnxF6uq\nzrcs9/Blsetcr0laP8tetz/JnwFvAJ8Cbhkr1/W9qrphid913X5pxqa2bn+Sy5K85+3nwO8CP8Zy\nXdJcW3LmT3I98K1hcyPw11X1+SRXAgeAa4GXGf2p78wSn+XML83YpDO/5bqkBWO5LkkXZPilpgy/\n1JThl5oy/FJThl9qyvBLTRl+qSnDLzVl+KWmDL/UlOGXmjL8UlOGX2rK8EtNGX6pKcMvNWX4paYm\nCn+Sy5M8nOS5JMeS3JTkiiSHkjw/PG6adWclTc+kM/+XgO9U1fuBDwDHsFyXNNcmWb33vcAzwPU1\n9uYkx3HdfumiM80FPK8DfgJ8NcnTSfYN6/dbrkuaY5OEfyPwQeDLVbUD+DlnHeIPRwTnLdeV5EiS\nI6vtrKTpmST8J4ATVfX4sP0woy+DU8PhPsPj6XP9clXtraqdY6W9JV0Elgx/Vb0GvJLk7fP5W4Gj\nWK5LmmsTVexJciOwD7gUeBH4BKMvDst1SRcZy3VJTVmuS9IFGX6pKcMvNWX4paYMv9SU4ZeaMvxS\nU4ZfasrwS00Zfqkpwy81Zfilpgy/1JThl5oy/FJThl9qyvBLTRl+qaklw5/khiTPjP28nuR+y3VJ\n821Za/gl2QC8Cvwm8GngTFU9mGQPsKmqPrvE77uGnzRjs1rD71bg36rqZeAuYP/Qvh/42DI/S9I6\nWm747wG+Pjy3XJc0xyYOf5JLgTuBvz37Nct1SfNnOTP/R4CnqurUsG25LmmOLSf89/LOIT9Yrkua\na5OW67oM+A/g+qr6n6HtSizXJV10LNclNWW5LkkXZPilpgy/1JThl5oy/FJThl9qyvBLTRl+qSnD\nLzVl+KWmDL/UlOGXmjL8UlOGX2rK8EtNGX6pKcMvNWX4paYMv9SU4ZeaMvxSUxvXeH//Cfx8eFxE\n72Mxx+a45sevT/rGNV26GyDJkUWt3rOoY3Nci8nDfqkpwy81tR7h37sO+1wrizo2x7WA1vycX9LF\nwcN+qak1DX+S25McT/JCkj1rue9pSnJNkseSHE3ybJLdQ/sVSQ4leX543LTefV2JJBuSPJ3k4LC9\nKOO6PMnDSZ5LcizJTYsytpVYs/An2QD8BfARYDtwb5Lta7X/KXsT+OOq2g78FvDpYSx7gMNVtQ04\nPGzPo93AsbHtRRnXl4DvVNX7gQ8wGuOijG35qmpNfoCbgEfHth8AHlir/c94bN8GbgOOA5uHts3A\n8fXu2wrGsoVRCD4MHBzaFmFc7wX+neE611j73I9tpT9redh/NfDK2PaJoW2uJdkK7AAeB66qqpPD\nS68BV61Tt1bji8BngLfG2hZhXNcBPwG+OpzS7EtyGYsxthXxgt8qJHk38E3g/qp6ffy1Gk0lc/Wn\nlCR3AKer6snzvWcexzXYCHwQ+HJV7WB0m/kvHOLP8dhWZC3D/ypwzdj2lqFtLiW5hFHw/6qq/m5o\nPpVk8/D6ZuD0evVvhT4E3JnkJeAbwIeTfI35HxeMjjRPVNXjw/bDjL4MFmFsK7KW4X8C2JbkuiSX\nAvcAj6zh/qcmSYCvAMeq6gtjLz0C3Dc8v4/RtYC5UVUPVNWWqtrK6N/nu1X1ceZ8XABV9RrwSpIb\nhqZbgaMswNhWak1v8knyUUbnlBuAh6rq82u28ylKcjPwfeBHvHNu/DlG5/0HgGuBl4G7q+rMunRy\nlZLcAvxJVd2R5EoWYFxJbgT2AZcCLwKfYDQBzv3YVsI7/KSmvOAnNWX4paYMv9SU4ZeaMvxSU4Zf\nasrwS00Zfqmp/wMvLV/L8v3Y4AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115e54978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "remove background\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACy5JREFUeJzt3W+oX4V9x/H3Z1HpaLeq1YVgdNcHYgkDky44xTI2NSPt\nxO6RKHRIKfikGyodRftsDwY+atoHo1DUTqhr56xSkWJx1rINijNWt9bELM5FjKiJ7Yqdgw3b7x78\nTultapJz//3uPff7fsHl/s753XDOIbzv+f1+OTnfVBWS+vm19d4BSevD+KWmjF9qyvilpoxfasr4\npaaMX2pqRfEn2ZvkUJIXk9yxWjslae1luRf5JNkC/DuwBzgKPA3cVFUHVm/3JK2VM1bwZy8HXqyq\nlwCSfA34GHDS+M8777xaWFhYwSYlncqRI0d48803M+ZnVxL/BcAri5aPAr93qj+wsLDA/v37V7BJ\nSaeye/fu0T+75h/4Jbklyf4k+48fP77Wm5M00krifxW4cNHy9mHdL6mqL1XV7qraff75569gc5JW\n00rifxq4JMnFSc4CbgQeWZ3dkrTWlv2ev6reSfJnwLeALcC9VfX8qu2ZpDW1kg/8qKpvAt9cpX2R\nNEde4Sc1ZfxSU8YvNWX8UlPGLzVl/FJTxi81tW7x79u3j3379q3X5qX2PPNLTRm/1JTxS00Zv9TU\niv5jz0rcfvvt67VpSXjml9oyfqkp45eaMn6pKeOXmjpt/EnuTXIsyQ8WrTs3yeNJDg/fz1nb3ZS0\n2sac+f8G2HvCujuAJ6rqEuCJYVnShJw2/qr6R+BHJ6z+GHDf8Pg+4E9Web8krbHlvuffWlWvDY9f\nB7au0v5ImpMVf+BXszG/Jx3167guaWNabvxvJNkGMHw/drIfdFyXtDEtN/5HgJuHxzcD31id3ZE0\nL2P+qe+rwHeBS5McTfJJ4C5gT5LDwLXDsqQJOe3/6quqm07y1DWrvC+S5sgr/KSmjF9qyvilpoxf\nasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2pqzD38Lkzy\nZJIDSZ5Pcuuw3pFd0oSNOfO/A3y6qnYAVwCfSrIDR3ZJkzZmXNdrVfW94fFPgIPABTiyS5q0Jb3n\nT7IA7AKewpFd0qSNjj/J+4CvA7dV1VuLnzvVyC7HdUkb06j4k5zJLPz7q+qhYfWokV2O65I2pjGf\n9ge4BzhYVZ9b9JQju6QJO+3EHuAq4E+B7yd5blj3WWYjuh4Yxne9DNywNrsoaS2MGdf1z0BO8rQj\nu6SJ8go/qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqkp\n45eaMn6pKeOXmjJ+qakxd+99T5J/SfKvw6y+vxzWO6tPmrAxZ/7/Ba6uqsuAncDeJFfgrD5p0sbM\n6quq+u9h8czhq3BWnzRpYyf2bBnu2X8MeLyqRs/qc1yXtDGNir+qflpVO4HtwOVJfueE5086q89x\nXRIk+ZWv9bakT/ur6sfAk8BeRs7qk7Qxjfm0//wkZw+Pfx3YA7yAs/qkSRszq28bcF+SLcx+WTxQ\nVY8m+S7O6pMma8ysvn8Ddr3L+h/irD5psrzCT2rK+KWmjF9qyvilpoxfasr4paaMX2rK+KWmjF9q\nyvilpoxfasr4paaMX2rK+KWmjF9qyvilpoxfamrMbbwkrdDsBtcby+gz/3Dv/meTPDosO65LmrCl\nvOy/FTi4aNlxXdKEjZ3Ysx34Y+DuRasd1yVN2Ngz/+eBzwA/W7TOcV3ShI0Z2nEdcKyqnjnZzziu\nS5qeMZ/2XwVcn+SjwHuA30zyFYZxXVX1muO6pOkZM6L7zqraXlULwI3At6vq4ziuS5q0lVzkcxew\nJ8lh4NphWdJELOkin6r6DvCd4bHjuqQJ8/JeqSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqkp\n45eaMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmhp1G68kR4CfAD8F3qmq3UnOBf4OWACOADdU\n1X+tzW5KWm1LOfP/YVXtrKrdw7LjuqQJW8nLfsd1SRM2Nv4C/iHJM0luGdaNGtclaWMae+vuD1fV\nq0l+C3g8yQuLn6yqSvKu47qGXxa3AFx00UUr2llJq2fUmb+qXh2+HwMeBi5nGNcFcKpxXc7qkzam\nMYM635vkN37+GPgj4Ac4rkuatDEv+7cCDyf5+c//bVU9luRp4IEknwReBm5Yu92UtNpOG39VvQRc\n9i7rHdclTZhX+ElNGb/UlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtNGb/U\nlPFLTRm/1JTxS00Zv9TUqPiTnJ3kwSQvJDmY5Mok5yZ5PMnh4fs5a72zklbP2DP/F4DHquqDzO7n\ndxDHdUmTNubW3e8Hfh+4B6Cq/q+qfozjuqRJG3Pmvxg4Dnw5ybNJ7h7u3++4LmnCxsR/BvAh4ItV\ntQt4mxNe4ldVMZvn9yuS3JJkf5L9x48fX+n+SlolY+I/ChytqqeG5QeZ/TJwXJc0YaeNv6peB15J\ncumw6hrgAI7rkiZt7JTePwfuT3IW8BLwCWa/OBzXJU3UqPir6jlg97s85bguaaK8wk9qyvilpoxf\nasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2rK+KWmxgzt\nuDTJc4u+3kpym+O6pGkbc/feQ1W1s6p2Ar8L/A/wMI7rkiZtqS/7rwH+o6pexnFd0qQtNf4bga8O\njx3XJU3Y6PiHe/ZfD/z9ic85rkuanqWc+T8CfK+q3hiWHdclTdhS4r+JX7zkB8d1SZM2Kv5hJPce\n4KFFq+8C9iQ5DFw7LEuaiLHjut4GPnDCuh/iuC5psrzCT2rK+KWmjF9qyvilpoxfasr4paaMX2rK\n+KWmjF9qyvilpoxfasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2rK+KWmjF9qKrNhO3PaWHIc\neBt4c24bna/z2JzH5nFNx29X1ajpOHONHyDJ/qraPdeNzslmPTaPa3PyZb/UlPFLTa1H/F9ah23O\ny2Y9No9rE5r7e35JG4Mv+6Wm5hp/kr1JDiV5Mckd89z2akpyYZInkxxI8nySW4f15yZ5PMnh4fs5\n672vy5FkS5Jnkzw6LG+W4zo7yYNJXkhyMMmVm+XYlmNu8SfZAvw18BFgB3BTkh3z2v4qewf4dFXt\nAK4APjUcyx3AE1V1CfDEsDxFtwIHFy1vluP6AvBYVX0QuIzZMW6WY1u6qprLF3Al8K1Fy3cCd85r\n+2t8bN8A9gCHgG3Dum3AofXet2Ucy3ZmEVwNPDqs2wzH9X7gPxk+51q0fvLHttyveb7svwB4ZdHy\n0WHdpCVZAHYBTwFbq+q14anXga3rtFsr8XngM8DPFq3bDMd1MXAc+PLwlubuJO9lcxzbsviB3wok\neR/wdeC2qnpr8XM1O5VM6p9SklwHHKuqZ072M1M8rsEZwIeAL1bVLmaXmf/SS/wJH9uyzDP+V4EL\nFy1vH9ZNUpIzmYV/f1U9NKx+I8m24fltwLH12r9lugq4PskR4GvA1Um+wvSPC2avNI9W1VPD8oPM\nfhlshmNblnnG/zRwSZKLk5wF3Ag8Msftr5okAe4BDlbV5xY99Qhw8/D4ZmafBUxGVd1ZVduraoHZ\n38+3q+rjTPy4AKrqdeCVJJcOq64BDrAJjm255v2/+j7K7D3lFuDeqvqruW18FSX5MPBPwPf5xXvj\nzzJ73/8AcBHwMnBDVf1oXXZyhZL8AfAXVXVdkg+wCY4ryU7gbuAs4CXgE8xOgJM/tuXwCj+pKT/w\nk5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqmp/wcJ7io9i9vaaAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115dae518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "set everything that's not black to white\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACyZJREFUeJzt3VGIZYV9x/Hvr6uSkrRRo10WVzs+iGEpuJsuVjGUVt2y\nScX0SRRSJAR8SYtCStC89aHgU0geSiCoqRCb1BolIsGwNYa2EKxrtE3cdbvWrrii7po0mFpoMfn3\n4Z6QycbdPTNz586e+X8/MMw9585yzkG+c+69njn/VBWS+vm1jd4BSRvD+KWmjF9qyvilpoxfasr4\npaaMX2pqTfEn2ZvkUJIXk9w5r52StP6y2ot8kmwB/h3YAxwFngZuqaoD89s9SevlrDX82yuBF6vq\nJYAkXwM+Bpw0/gsuuKCWlpbWsElJp3LkyBHefPPNjPnZtcR/EfDKsuWjwO+d6h8sLS2xf//+NWxS\n0qns3r179M+u+wd+SW5Lsj/J/uPHj6/35iSNtJb4XwUuXra8fVj3S6rqS1W1u6p2X3jhhWvYnKR5\nWkv8TwOXJbk0yTnAzcCj89ktSett1e/5q+qdJH8GfAvYAtxXVc/Pbc8krau1fOBHVX0T+Oac9kXS\nAnmFn9SU8UtNGb/UlPFLTRm/1JTxS00Zv9TUhsWfhGTUHx9JWgee+aWmjF9qyvilpoxfampNf9iz\nFg4IlTaWZ36pKeOXmjJ+qSnjl5oyfqmp08af5L4kx5L8YNm685PsS3J4+H7e+u6mpHkbc+b/G2Dv\nCevuBJ6oqsuAJ4ZlSRNy2vir6h+BH52w+mPA/cPj+4E/mfN+SVpnq33Pv7WqXhsevw5sndP+SFqQ\nNX/gV7NL9U56uZ7juqQz02rjfyPJNoDh+7GT/aDjuqQz02rjfxS4dXh8K/CN+eyOpEUZ87/6vgp8\nF7g8ydEknwTuBvYkOQxcPyxLmpDT/lVfVd1ykqeum/O+SFogr/CTmjJ+qSnjl5oyfqkp45eaMn6p\nKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pqTH38Ls4yZNJDiR5Psnt\nw3pHdkkTNubM/w7w6araAVwFfCrJDhzZJU3amHFdr1XV94bHPwEOAhfhyC5p0lb0nj/JErALeApH\ndkmTNjr+JO8Dvg7cUVVvLX/uVCO7HNclnZlGxZ/kbGbhP1BVDw+rR43sclyXdGYa82l/gHuBg1X1\nuWVPObJLmrDTTuwBrgH+FPh+kueGdZ9lNqLrwWF818vATeuzi5LWw5hxXf8M5CRPO7JLmiiv8JOa\nMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oy\nfqkp45eaGnP33vck+Zck/zrM6vvLYb2z+qQJG3Pm/1/g2qq6AtgJ7E1yFc7qkyZtzKy+qqr/HhbP\nHr4KZ/VJkzZ2Ys+W4Z79x4B9VTV6Vp/juqQz06j4q+qnVbUT2A5cmeR3Tnj+pLP6HNclQZJf+dpo\nK/q0v6p+DDwJ7GXkrD5JZ6Yxn/ZfmOTc4fGvA3uAF3BWnzRpY2b1bQPuT7KF2S+LB6vqsSTfxVl9\n0mSNmdX3b8Cud1n/Q5zVJ02WV/hJTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtN\nGb/UlPFLTRm/1JTxS00Zv9SU8UtNjbmNl6Q1mt3g+swy+sw/3Lv/2SSPDcuO65ImbCUv+28HDi5b\ndlyXNGFjJ/ZsB/4YuGfZasd1SRM29sz/eeAzwM+WrXNclzRhY4Z23AAcq6pnTvYzjuuSpmfMp/3X\nADcm+SjwHuA3k3yFYVxXVb3muC5pesaM6L6rqrZX1RJwM/Dtqvo4juuSJm0tF/ncDexJchi4fliW\nNBErusinqr4DfGd47LguacK8vFdqyvilpoxfasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2rK\n+KWmjF9qyvilpoxfasr4paaMX2rK+KWmRt3GK8kR4CfAT4F3qmp3kvOBvwOWgCPATVX1X+uzm5Lm\nbSVn/j+sqp1VtXtYdlyXNGFrednvuC5pwsbGX8A/JHkmyW3DulHjuiSdmcbeuvvDVfVqkt8C9iV5\nYfmTVVVJ3nVc1/DL4jaASy65ZE07K2l+Rp35q+rV4fsx4BHgSoZxXQCnGtflrD7pzDRmUOd7k/zG\nzx8DfwT8AMd1SZM25mX/VuCRJD//+b+tqseTPA08mOSTwMvATeu3m5Lm7bTxV9VLwBXvst5xXdKE\neYWf1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtN\nGb/UlPFLTY2KP8m5SR5K8kKSg0muTnJ+kn1JDg/fz1vvnZU0P2PP/F8AHq+qDzK7n99BHNclTdqY\nW3e/H/h94F6Aqvq/qvoxjuuSJm3Mmf9S4Djw5STPJrlnuH+/47qkCRsT/1nAh4AvVtUu4G1OeIlf\nVcVsnt+vSHJbkv1J9h8/fnyt+ytpTsbEfxQ4WlVPDcsPMftl4LguacJOG39VvQ68kuTyYdV1wAEc\n1yVN2tgpvX8OPJDkHOAl4BPMfnE4rkuaqFHxV9VzwO53ecpxXdJEeYWf1JTxS00Zv9SU8UtNGb/U\nlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTY0Z2nF5kueWfb2V\n5A7HdUnTNubuvYeqamdV7QR+F/gf4BEc1yVN2kpf9l8H/EdVvYzjuqRJW2n8NwNfHR47rkuasNHx\nD/fsvxH4+xOfc1yXND0rOfN/BPheVb0xLDuuS5qwlcR/C794yQ+O65ImbVT8w0juPcDDy1bfDexJ\nchi4fliWNBFjx3W9DXzghHU/xHFd0mR5hZ/UlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTx\nS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1FRmw3YWtLHkOPA28ObCNrpY\nF7A5j83jmo7frqpR03EWGj9Akv1VtXuhG12QzXpsHtfm5Mt+qSnjl5raiPi/tAHbXJTNemwe1ya0\n8Pf8ks4MvuyXmlpo/En2JjmU5MUkdy5y2/OU5OIkTyY5kOT5JLcP689Psi/J4eH7eRu9r6uRZEuS\nZ5M8NixvluM6N8lDSV5IcjDJ1Zvl2FZjYfEn2QL8NfARYAdwS5Idi9r+nL0DfLqqdgBXAZ8ajuVO\n4Imqugx4YlieotuBg8uWN8txfQF4vKo+CFzB7Bg3y7GtXFUt5Au4GvjWsuW7gLsWtf11PrZvAHuA\nQ8C2Yd024NBG79sqjmU7swiuBR4b1m2G43o/8J8Mn3MtWz/5Y1vt1yJf9l8EvLJs+eiwbtKSLAG7\ngKeArVX12vDU68DWDdqttfg88BngZ8vWbYbjuhQ4Dnx5eEtzT5L3sjmObVX8wG8NkrwP+DpwR1W9\ntfy5mp1KJvW/UpLcAByrqmdO9jNTPK7BWcCHgC9W1S5ml5n/0kv8CR/bqiwy/leBi5ctbx/WTVKS\ns5mF/0BVPTysfiPJtuH5bcCxjdq/VboGuDHJEeBrwLVJvsL0jwtmrzSPVtVTw/JDzH4ZbIZjW5VF\nxv80cFmSS5OcA9wMPLrA7c9NkgD3Ager6nPLnnoUuHV4fCuzzwImo6ruqqrtVbXE7L/Pt6vq40z8\nuACq6nXglSSXD6uuAw6wCY5ttRb9V30fZfaecgtwX1X91cI2PkdJPgz8E/B9fvHe+LPM3vc/CFwC\nvAzcVFU/2pCdXKMkfwD8RVXdkOQDbILjSrITuAc4B3gJ+ASzE+Dkj201vMJPasoP/KSmjF9qyvil\npoxfasr4paaMX2rK+KWmjF9q6v8BJDIok+zbVjUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1160a2ef0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# crop\n",
    "print(\"crop\")\n",
    "processed_frame = frame[35:195]\n",
    "display_frame(processed_frame)\n",
    "# reduce image size\n",
    "print(\"reduce image size\")\n",
    "processed_frame = processed_frame[::2, ::2, :]\n",
    "display_frame(processed_frame)\n",
    "# remove color\n",
    "print(\"remove color\")\n",
    "processed_frame = processed_frame[:, :, 0]\n",
    "display_frame(processed_frame)\n",
    "# remove background\n",
    "print(\"remove background\")\n",
    "processed_frame[processed_frame == 144] = 0\n",
    "processed_frame[processed_frame == 109] = 0\n",
    "display_frame(processed_frame)\n",
    "# set everything that's not black to white\n",
    "print(\"set everything that's not black to white\")\n",
    "processed_frame[processed_frame != 0] = 1\n",
    "display_frame(processed_frame)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now the new dimensions and size (i.e. when flattening the matrix into a vector) are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dimensions: (80, 80)\n",
      "Size: 6400\n"
     ]
    }
   ],
   "source": [
    "print(\"Dimensions: {0}\".format(processed_frame.shape))\n",
    "print(\"Size: {0}\".format(processed_frame.size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This new frame size is much more computationally managable which will make training a neural network a bit easier. Let's make a function for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "unprocessed frame\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEhVJREFUeJztnXvMJmV5xn9Xd5dDOIRjt6tg3TULCat2lQ2aaCmNJyC2\nSNtYiDGLGlcSa8Rq4iJN3dgQLRX8Q1ItCBWrgsRDpY2EgJGqiSiLWYGFrqwcInTdFfAALaC7XP3j\nnU9f1u/wzs473zMzXL/kyz3vM/fM/dz75Z7rmdn57le2CSHsO79XegIh9J0UUQgNSRGF0JAUUQgN\nSRGF0JAUUQgNaa2IJJ0qaZuk7ZI2thUnhNKojf8nkrQE+CHwGuBB4FbgbNt3TT1YCIVpS4lOArbb\nvtf2r4BrgDNaihVCUZa2dN7nAj8e+/wg8LK5nCXNK4e/v/yAKU0rhMnZtfPJh20fvZBfW0W0IJI2\nABsADjl0Geece1ypqczK209ZU/uYy2/e2sJM+s+TT91Q+5gD9n9dCzOpx8cv2vrAJH5tLeceAo4d\n+3xMNfYbbF9me53tdQceuKSlaYTQPm0V0a3AakkrJe0HnAVc11KsEIrSynLO9m5JfwPcACwBrrSd\ntU4YJK3dE9n+GvC1ts6/2Mx2v7Mv901h9vudfblv6gp5YyGEhqSIQmhIiiiEhqSIQmhIiiiEhqSI\nQmhIiiiEhqSIQmhIiiiEhqSIQmhIiiiEhqSIQmhIsT/K6xt52XR69Pll09mIEoXQkBRRCA1JEYXQ\nkNwTzUGajkyPLjQdaZN9ViJJx0r6hqS7JG2V9O5qfJOkhyRtqX5On950Q+geTZRoN/Be29+XdAhw\nm6Qbq30fs/3R5tMLofvscxHZ3gHsqLYfk3Q3o6aNtXl8925u2fnIvk4lhKJM5cGCpOcDLwG+Ww29\nS9Ltkq6UdPg0YoTQVRoXkaSDgS8B59n+JfAJYBWwlpFSXTzHcRskbZa0efeTTzedRgjFaFREkpYx\nKqDP2f4ygO2dtvfYfhq4nFFz+99hvAPq0gPypD30lyZP5wRcAdxt+5Kx8RVjbmcCd+779ELoPk2e\nzr0CeDNwh6Qt1dgHgLMlrQUM3A+8o9EMQ+g4TZ7OfRvQLLsG0/U0hEnIzUgIDUkRhdCQFFEIDenE\nC6gHL13Ky5cfWXoaITyDW/nJRH6dUKJdlf14TbsvxzSxQ41FTd++5dfk32QSZM/7ncOLwvI/ONB/\nvf4FpacRwjP4+EVbb7O9biG/KFENO9RY1PTtW35RohAKESVqwQ41FjV9+5Zf20rUiSIKoc9kORfC\nHGQ514Idaixq+vYtv7aXc1GiEOYgStSCHWosavr2Lb8oUQiF6KUSXbrzEW7Z+Uiunosci5q+fcvv\nWaVEM22z8jJq6AKLokSS7pd0R9XpdHM1doSkGyXdU9kFW2bNKNHmyubqGSXqQqxJaaREku4H1tl+\neGzsIuBR2x+RtBE43Pb75ztPlCh0kZL3RGcAV1XbVwFvWOiAGSW6dfmRvHz5kbl6LnIsavr2Lb+u\nK9F9wC+APcC/2L5M0s9tH1btF/Czmc9zkadzoYsslhK90vZa4DTgnZJOHt/pUYXOWqXjHVB3PbFn\nNOlqX66eUaIuxJqUqT2dk7QJeBx4O3CK7R1VI8ebbR8/37FRotBFWlciSQdVX6mCpIOA1zLqdnod\nsL5yWw98daFz5Y2FsrGo6du3/DqrRJJWAV+pPi4FPm/7QklHAtcCzwMeAN5o+9H5zhUlCl2kdSWy\nfa/tP6p+1ti+sBp/xParbK+2/eqFCgiiRKVjUdO3b/l1VommSZQodJFevjuXq2eZWNT07Vt+UaIQ\nChElasEONRY1ffuWX5QohEJEiVqwQ41FTd++5RclCqEQUaIW7FBjUdO3b/lFiUIoRJSoBTvUWNT0\n7Vt+bStRJ4oohD6T5VwIc5DlXAt2qLGo6du3/NpezkWJQpiDKFELdqixqOnbt/yiRCEUIkrUgh1q\nLGr69i2/ziqRpOOBL4wNrQL+HjiMUbOSn1bjH7D9tfnOFSUKXWQx/jx8m+21VcusE4H/47c9Fz42\ns2+hAoIoUelY1PTtW36dVaJnnER6LfBB26+YaZ1l+6OTHh8lCl1kse+JzgKuHvv8Lkm3S7qyTkP7\nXD3LxKKmb9/y67wSSdoP+B9gje2dkpYDDzPqfPoPwArbb53luA3ABoBDDl124jnnHtdoHiFMm8VU\notOA79veCWB7p+09tp8GLgdOmu0g25fZXmd73WMHLhlNembyE9p9OWboV88o0fRiTco0lOga4Abb\n/1p9XmF7R7X9HuBlts+a7xy5JwpdZLG+5Osg4DXAl8eGL6q++Ot24E+B9yx0ntwTlY1FTd++5dd5\nJZoGUaLQRfLGQgt2qLGo6du3/KJEIRQiStSCHWosavr2Lb8oUQiFiBK1YIcai5q+fcsvShRCIaJE\nLdihxqKmb9/yixKFUIgoUQt2qLGo6du3/KJEIRQiStSCHWosavr2Lb+2lagTRRRCn8lyLoQ5yHKu\nBTvUWNT07Vt+bS/nokQhzEGUqAU71FjU9O1bflGiECbgyadueMbnA/Z/XeNzTk2JqrZXuyTdOTZ2\nhKQbJd1T2cPH9p0vabukbZImyiRKVDYWNX27mN+n9rKdUiJJJwOPA5+x/cJq7CLgUdsfkbQRONz2\n+yWdwKj/3EnAc4CbgONs75kvRpQoNKXTSmT7m8Cjew2fAVxVbV8FvGFs/BrbT9m+D9jOHC2zxokS\nlY1FTd8u5tdpJQKQ9HzgP8eU6Oe2D6u2BfzM9mGSLgVusf3Zat8VwPW2vzjLOdO8MUyNTivRQnhU\nhbWfTqR5Y3diUdO3i/n1UYm2AafY3iFpBXCz7eMlnQ9g+8OV3w3AJtvfme/8uScKTemjEl0HrK+2\n1wNfHRs/S9L+klYCq4HvLXSy3BOVjUVN3y7m12klknQ1cApwFLAT+CDw78C1wPOAB4A32n608r8A\neCuwGzjP9vULTSJKFJrSaSWyfbbtFbaX2T7G9hW2H7H9Kturbb96poAq/wttv8D28ZMUEESJSsei\npm8X8+u0Ei0GUaLQlE4r0WIQJSobi5q+XcwvShQlCg2JElX22Xb17Eosavp2Mb8oUZQoNCRKVNln\n29WzK7Go6dvF/KJEUaLQQaJELdihxqKmb9/yixKFUIgoUQt2qLGo6du3/KJEIRQiStSCHWosavr2\nLb8oUQiF6JUShdBnOlFEWc6VjUVN377ll+VcCIXo1XIuSlQ2FjV9+5ZfcSWSdCXwemDXWKOSfwL+\nDPgV8CPgLbZ/XjU0uRvYVh1+i+1zF5pElCh0kWkq0aeBU/cauxF4oe0XAz8Ezh/b9yPba6ufBQsI\nokSlY1HTt2/5FVci+N2WWXvtOxP4K9tvms9vPqJEoYss5j3RW4HxhiQrJW2R9F+S/niugyRtkLRZ\n0uZdT4xadefqWSYWNX37ll+nlahqj7UO+AvblrQ/cLDtRySdyKi11hrbv5zv/FGi0EVaVyJJ5zB6\n4PCmqpUwVSP7R6rt2xg9dFiwyXbuicrGoqZv3/LrpBJJOhW4BPgT2z8d8zua0Veu7JG0CvgW8KLx\nvnSzESUKXWSaX/J1NfAd4HhJD0p6G3ApcAhwY3X/88nK/WTgdklbgC8C5y5UQBAlKh2Lmr59y68T\nStQ2UaLQRfLGQgt2qLGo6du3/KJEIRQiStSCHWosavr2Lb8oUQiFiBK1YIcai5q+fcsvShRCIaJE\nLdihxqKmb9/yixKFUIgoUQt2qLGo6du3/KJEIRQiStSCHWosavr2Lb8oUQiFiBK1YIcai5q+fcsv\nShRCIXqlRCH0mU4UUZZzZWNR07dv+WU5F0Ihpvnn4VdK2iXpzrGxTZIeqv40fIuk08f2nS9pu6Rt\nkib6HvQoUdlY1PTtW37FlUjSycDjwGfGGpVsAh63/dG9fE8ArgZOAp4D3AQcZ3vPfDGiRKGLTE2J\nbH8TWLDZSMUZwDVV66z7gO2MCmpeokRlY1HTt2/5FVcimLVl1ibgLcAvgM3Ae23/TNKljJrYf7by\nuwK43vYXZznnBmADwCGHLjvxnHMXbE8XwqLS9iPuTwCrgLXADuDiuiewfZntdbbXPXbgEiBXzyhR\nt2JNyjQa2v9mn6TzAWx/uNp3A7DJ9nfmO3/uiUIXaVWJJK0Y+3gmMPPk7jrgLEn7S1oJrAa+t9D5\nck9UNhY1ffuWX3ElqjqgngIcBewEPlh9XgsYuB94h+0dlf8FjL4pYjdwnu3rf+ekexElCl1kmk/n\nzra9wvYy28fYvsL2m22/yPaLbf/5TAFV/hfafoHt4ycpIIgSlY5FTd++5VdciRaDKFHoIr16ATVK\nVDYWNX37ll+UKIRCRIlasEONRU3fvuUXJQqhEFGiFuxQY1HTt2/5RYlCKESUqAU71FjU9O1bflGi\nEAoRJWrBDjUWNX37ll+UKIRCRIlasEONRU3fvuUXJQqhEFGiFuxQY1HTt2/5RYlCKESvlCiEPrOv\nzRu/MNa48X5JW6rx50t6YmzfJyeZRJZzZWNR07dv+RVfzs3WvHGv/RcDv7D9ofkamsxHlnOhiyxK\n80ZJAt7IqOvpPhMlKhuLmr59y6+4EsHcLbMqlbpkplorv63APYwaO/6d7W8tdP4oUegii/Vg4Wye\nqUI7gOfZXgv8LfB5SYfOdqCkDZI2S9q864lRq+5cPcvEoqZv3/LrrBJJWgo8BJxo+8E5jrsZeJ/t\nzfOdP0oUushiKNGrgf8eLyBJR0taUm2vYtS88d6FTpR7orKxqOnbt/yKK9FszRttXyHp04ya139y\nzPcvgQ8Bvwaernz/Y6FJRIlCF2m1eWM1fs54AVVjX7K9xvZa2y+dpIAgSlQ6FjV9+5ZfcSVaDKJE\noYv06rWfKFHZWNT07Vt+UaIQChElasEONRY1ffuWX5QohEJEiVqwQ41FTd++5RclCqEQUaIW7FBj\nUdO3b/lFiUIoRJSoBTvUWNT07Vt+UaIQChElasEONRY1ffuWX5QohEJEiVqwQ41FTd++5fesUKKD\njlrmNa8/qvQ0QngGt171k/4oUQh9JkUUQkMm6YB6rKRvSLpL0lZJ767Gj5B0o6R7Knv42DHnS9ou\naZuk17WZQAilmUSJdgPvtX0C8HLgnZJOADYCX7e9Gvh69Zlq31nAGuBU4J9nmpeEMEQm6bGww/b3\nq+3HgLuB5wJnAFdVblcBb6i2zwCusf2U7fuA7cBJ0554CF2h1j1R1X/uJcB3geW2d1S7fgIsr7af\nC/x47LAHq7EQBsnERSTpYOBLwHm2fzm+z6Pn5LWelY93QN395NN1Dg2hU0xURJKWMSqgz9n+cjW8\nU9KKav8Kfvt/pg8Bx44dfkw19gxsX2Z7ne11Sw/IQ8LQXyZ5OifgCuBu25eM7boOWF9trwe+OjZ+\nlqT9Ja1k1AX1e9ObcgjdYukEPq8A3gzcMfNlXsAHgI8A10p6G/AAo69YwfZWSdcCdzF6svdO23um\nPvMQOsKCRWT724Dm2P2qOY65ELiwwbxC6A25GQmhISmiEBqSIgqhISmiEBqSIgqhIZ34ozxJPwX+\nF3i49FymyFEMJ58h5QKT5/OHto9eyKkTRQQgafMkf0XYF4aUz5Bygennk+VcCA1JEYXQkC4V0WWl\nJzBlhpTPkHKBKefTmXuiEPpKl5QohF5SvIgknVo1NNkuaWPp+ewLku6XdIekLZI2V2NzNnLpGpKu\nlLRL0p1jY71tRDNHPpskPVT9jrZIOn1sX7N8bBf7AZYAPwJWAfsBPwBOKDmnfczjfuCovcYuAjZW\n2xuBfyw9z3nmfzLwUuDOheYPnFD9nvYHVla/vyWlc5ggn03A+2bxbZxPaSU6Cdhu+17bvwKuYdTo\nZAjM1cilc9j+JvDoXsO9bUQzRz5z0Tif0kU0lKYmBm6SdJukDdXYXI1c+sIQG9G8S9Lt1XJvZnna\nOJ/SRTQUXml7LXAao758J4/v9Gjd0NvHoH2ff8UnGN02rAV2ABdP68Sli2iipiZdx/ZDld0FfIXR\ncmCuRi59oVEjmq5he6ftPbafBi7nt0u2xvmULqJbgdWSVkraj1Hn1OsKz6kWkg6SdMjMNvBa4E7m\nbuTSFwbViGbmglBxJqPfEUwjnw48STkd+CGjpyIXlJ7PPsx/FaOnOz8Ats7kABzJqL3yPcBNwBGl\n5zpPDlczWuL8mtE9wdvmmz9wQfX72gacVnr+E+bzb8AdwO1V4ayYVj55YyGEhpRezoXQe1JEITQk\nRRRCQ1JEITQkRRRCQ1JEITQkRRRCQ1JEITTk/wF+0t4YeXNvSQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x115e5d9b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "processed frame\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACyZJREFUeJzt3VGIZYV9x/Hvr6uSkrRRo10WVzs+iGEpuJsuVjGUVt2y\nScX0SRRSJAR8SYtCStC89aHgU0geSiCoqRCb1BolIsGwNYa2EKxrtE3cdbvWrrii7po0mFpoMfn3\n4Z6QycbdPTNz586e+X8/MMw9585yzkG+c+69njn/VBWS+vm1jd4BSRvD+KWmjF9qyvilpoxfasr4\npaaMX2pqTfEn2ZvkUJIXk9w5r52StP6y2ot8kmwB/h3YAxwFngZuqaoD89s9SevlrDX82yuBF6vq\nJYAkXwM+Bpw0/gsuuKCWlpbWsElJp3LkyBHefPPNjPnZtcR/EfDKsuWjwO+d6h8sLS2xf//+NWxS\n0qns3r179M+u+wd+SW5Lsj/J/uPHj6/35iSNtJb4XwUuXra8fVj3S6rqS1W1u6p2X3jhhWvYnKR5\nWkv8TwOXJbk0yTnAzcCj89ktSett1e/5q+qdJH8GfAvYAtxXVc/Pbc8krau1fOBHVX0T+Oac9kXS\nAnmFn9SU8UtNGb/UlPFLTRm/1JTxS00Zv9TUhsWfhGTUHx9JWgee+aWmjF9qyvilpoxfampNf9iz\nFg4IlTaWZ36pKeOXmjJ+qSnjl5oyfqmp08af5L4kx5L8YNm685PsS3J4+H7e+u6mpHkbc+b/G2Dv\nCevuBJ6oqsuAJ4ZlSRNy2vir6h+BH52w+mPA/cPj+4E/mfN+SVpnq33Pv7WqXhsevw5sndP+SFqQ\nNX/gV7NL9U56uZ7juqQz02rjfyPJNoDh+7GT/aDjuqQz02rjfxS4dXh8K/CN+eyOpEUZ87/6vgp8\nF7g8ydEknwTuBvYkOQxcPyxLmpDT/lVfVd1ykqeum/O+SFogr/CTmjJ+qSnjl5oyfqkp45eaMn6p\nKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pqTH38Ls4yZNJDiR5Psnt\nw3pHdkkTNubM/w7w6araAVwFfCrJDhzZJU3amHFdr1XV94bHPwEOAhfhyC5p0lb0nj/JErALeApH\ndkmTNjr+JO8Dvg7cUVVvLX/uVCO7HNclnZlGxZ/kbGbhP1BVDw+rR43sclyXdGYa82l/gHuBg1X1\nuWVPObJLmrDTTuwBrgH+FPh+kueGdZ9lNqLrwWF818vATeuzi5LWw5hxXf8M5CRPO7JLmiiv8JOa\nMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oyfqkp45eaMn6pKeOXmjJ+qSnjl5oy\nfqkp45eaGnP33vck+Zck/zrM6vvLYb2z+qQJG3Pm/1/g2qq6AtgJ7E1yFc7qkyZtzKy+qqr/HhbP\nHr4KZ/VJkzZ2Ys+W4Z79x4B9VTV6Vp/juqQz06j4q+qnVbUT2A5cmeR3Tnj+pLP6HNclQZJf+dpo\nK/q0v6p+DDwJ7GXkrD5JZ6Yxn/ZfmOTc4fGvA3uAF3BWnzRpY2b1bQPuT7KF2S+LB6vqsSTfxVl9\n0mSNmdX3b8Cud1n/Q5zVJ02WV/hJTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtN\nGb/UlPFLTRm/1JTxS00Zv9SU8UtNjbmNl6Q1mt3g+swy+sw/3Lv/2SSPDcuO65ImbCUv+28HDi5b\ndlyXNGFjJ/ZsB/4YuGfZasd1SRM29sz/eeAzwM+WrXNclzRhY4Z23AAcq6pnTvYzjuuSpmfMp/3X\nADcm+SjwHuA3k3yFYVxXVb3muC5pesaM6L6rqrZX1RJwM/Dtqvo4juuSJm0tF/ncDexJchi4fliW\nNBErusinqr4DfGd47LguacK8vFdqyvilpoxfasr4paaMX2rK+KWmjF9qyvilpoxfasr4paaMX2rK\n+KWmjF9qyvilpoxfasr4paaMX2rK+KWmRt3GK8kR4CfAT4F3qmp3kvOBvwOWgCPATVX1X+uzm5Lm\nbSVn/j+sqp1VtXtYdlyXNGFrednvuC5pwsbGX8A/JHkmyW3DulHjuiSdmcbeuvvDVfVqkt8C9iV5\nYfmTVVVJ3nVc1/DL4jaASy65ZE07K2l+Rp35q+rV4fsx4BHgSoZxXQCnGtflrD7pzDRmUOd7k/zG\nzx8DfwT8AMd1SZM25mX/VuCRJD//+b+tqseTPA08mOSTwMvATeu3m5Lm7bTxV9VLwBXvst5xXdKE\neYWf1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtN\nGb/UlPFLTY2KP8m5SR5K8kKSg0muTnJ+kn1JDg/fz1vvnZU0P2PP/F8AHq+qDzK7n99BHNclTdqY\nW3e/H/h94F6Aqvq/qvoxjuuSJm3Mmf9S4Djw5STPJrlnuH+/47qkCRsT/1nAh4AvVtUu4G1OeIlf\nVcVsnt+vSHJbkv1J9h8/fnyt+ytpTsbEfxQ4WlVPDcsPMftl4LguacJOG39VvQ68kuTyYdV1wAEc\n1yVN2tgpvX8OPJDkHOAl4BPMfnE4rkuaqFHxV9VzwO53ecpxXdJEeYWf1JTxS00Zv9SU8UtNGb/U\nlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTY0Z2nF5kueWfb2V\n5A7HdUnTNubuvYeqamdV7QR+F/gf4BEc1yVN2kpf9l8H/EdVvYzjuqRJW2n8NwNfHR47rkuasNHx\nD/fsvxH4+xOfc1yXND0rOfN/BPheVb0xLDuuS5qwlcR/C794yQ+O65ImbVT8w0juPcDDy1bfDexJ\nchi4fliWNBFjx3W9DXzghHU/xHFd0mR5hZ/UlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1JTx\nS00Zv9SU8UtNGb/UlPFLTRm/1JTxS00Zv9SU8UtNGb/UlPFLTRm/1FRmw3YWtLHkOPA28ObCNrpY\nF7A5j83jmo7frqpR03EWGj9Akv1VtXuhG12QzXpsHtfm5Mt+qSnjl5raiPi/tAHbXJTNemwe1ya0\n8Pf8ks4MvuyXmlpo/En2JjmU5MUkdy5y2/OU5OIkTyY5kOT5JLcP689Psi/J4eH7eRu9r6uRZEuS\nZ5M8NixvluM6N8lDSV5IcjDJ1Zvl2FZjYfEn2QL8NfARYAdwS5Idi9r+nL0DfLqqdgBXAZ8ajuVO\n4Imqugx4YlieotuBg8uWN8txfQF4vKo+CFzB7Bg3y7GtXFUt5Au4GvjWsuW7gLsWtf11PrZvAHuA\nQ8C2Yd024NBG79sqjmU7swiuBR4b1m2G43o/8J8Mn3MtWz/5Y1vt1yJf9l8EvLJs+eiwbtKSLAG7\ngKeArVX12vDU68DWDdqttfg88BngZ8vWbYbjuhQ4Dnx5eEtzT5L3sjmObVX8wG8NkrwP+DpwR1W9\ntfy5mp1KJvW/UpLcAByrqmdO9jNTPK7BWcCHgC9W1S5ml5n/0kv8CR/bqiwy/leBi5ctbx/WTVKS\ns5mF/0BVPTysfiPJtuH5bcCxjdq/VboGuDHJEeBrwLVJvsL0jwtmrzSPVtVTw/JDzH4ZbIZjW5VF\nxv80cFmSS5OcA9wMPLrA7c9NkgD3Ager6nPLnnoUuHV4fCuzzwImo6ruqqrtVbXE7L/Pt6vq40z8\nuACq6nXglSSXD6uuAw6wCY5ttRb9V30fZfaecgtwX1X91cI2PkdJPgz8E/B9fvHe+LPM3vc/CFwC\nvAzcVFU/2pCdXKMkfwD8RVXdkOQDbILjSrITuAc4B3gJ+ASzE+Dkj201vMJPasoP/KSmjF9qyvil\npoxfasr4paaMX2rK+KWmjF9q6v8BJDIok+zbVjUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1162649b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def process_frame(frame, previously_processed_frame):\n",
    "    # crop\n",
    "    processed_frame = frame[35:195]\n",
    "    # reduce image size\n",
    "    processed_frame = processed_frame[::2, ::2, :]\n",
    "    # remove color\n",
    "    processed_frame = processed_frame[:, :, 0]\n",
    "    # remove background\n",
    "    processed_frame[processed_frame == 144] = 0\n",
    "    processed_frame[processed_frame == 109] = 0\n",
    "    # set everything that's not black to white\n",
    "    processed_frame[processed_frame != 0] = 1\n",
    "    # flatten frame\n",
    "    processed_frame = processed_frame.astype(np.float).ravel()\n",
    "\n",
    "    # subtract the previous frame from the current one so we are only processing\n",
    "    # on changes in the game\n",
    "    if previously_processed_frame is not None:\n",
    "        delta_frame = processed_frame - previously_processed_frame\n",
    "    else:\n",
    "        # the below flattens a frame of 0s\n",
    "        delta_frame = np.zeros(processed_frame.size)\n",
    "    # store the previous frame so we can subtract from it next time\n",
    "    previously_processed_frame = processed_frame\n",
    "    return delta_frame, previously_processed_frame\n",
    "\n",
    "print(\"unprocessed frame\")\n",
    "display_frame(frame)\n",
    "previously_processed_frame = None\n",
    "delta_frame, previously_processed_frame = process_frame(frame, previously_processed_frame)\n",
    "print(\"processed frame\")\n",
    "display_frame(previously_processed_frame.reshape((80, 80)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Set Up Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now to write a neural network model from scratch.\n",
    "\n",
    "The architecture of the neural network will have an input layer of 6400 inputs (because each frame of the pong game is 80 x 80 pixels), one hidden layer with a total of 200 neurons, and an output layer of one neuron (which will represent the probability of the AI moving the paddle up).\n",
    "\n",
    "We'll also need to set up some other hyperparameters like the number of games to play before we update the neural network, a paramter to control how fast we make changes to the weights of the neural network, and a variable to control the rate of decay when performing backpropagation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "num_hidden_layers = 200 # number of hidden layer neurons\n",
    "games_to_play = 10 # number of games to play before neural network update\n",
    "learning_rate = 1e-3 # how fast to make updates to weights\n",
    "backprop_rate = 0.99 # decay factor for RMSProp leaky sum of weight_gradients^2\n",
    "resume = True # resume using weights stored in pickle?\n",
    "render = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**Weights:**\n",
    "\n",
    "We'll want to initialize the weights of our network using the [\"Xavier\"](http://andyljones.tumblr.com/post/110998971763/an-explanation-of-xavier-initialization) method instead of just giving the weights random values from the standard normal distribution for the following reasons:\n",
    "* If the weights in a network start too small, then the signal shrinks as it passes through each layer until it’s too tiny to be useful.\n",
    "* If the weights in a network start too large, then the signal grows as it passes through each layer until it’s too massive to be useful.\n",
    "* Xavier initialization makes sure the weights are \"just right\", keeping the signal in a reasonable range of values through many layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# weights initialization\n",
    "input_layer_size = 80 * 80 # input dimensionality: 80x80 grid\n",
    "if resume:\n",
    "    weights = pickle.load(open('nn_weights_demo.pkl', 'rb'))\n",
    "else:\n",
    "    weights = {}\n",
    "    weights['W1'] = np.random.randn(num_hidden_layers,input_layer_size) / np.sqrt(input_layer_size) # \"Xavier\" initialization\n",
    "    weights['W2'] = np.random.randn(num_hidden_layers) / np.sqrt(num_hidden_layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**Forward propagation**:\n",
    "* $w^l_{jk}$: weight from the $k^{\\mathrm{th}}$ neuron in the $(l-1)^\\mathrm{th}$ layer to the $j^{\\mathrm{th}}$ neuron in the $l^{\\mathrm{th}}$ layer\n",
    "> The reason why we use this notation that $k$ is the input neuron and $j$ is the output neuron is to simplify the vectorized formula below for activating neurons and the intuition of saying \"apply the weight matrix to the activations\".\n",
    "* $a^{l}_j = \\sigma\\left( \\sum_k w^{l}_{jk} a^{l-1}_k \\right)$ which vectorized is $a^{l} = \\sigma(w^l a^{l-1})$\n",
    "* activation functions for hidden layers should use Relu instead of sigmoid\n",
    "* output layer activation should use sigmoid to provide a range from [0, 1] for probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1.0 / (1.0 + np.exp(-x)) # sigmoid \"squashing\" function to interval [0,1]\n",
    "\n",
    "def forward(frame):\n",
    "    hidden_layer = np.dot(weights['W1'], frame)\n",
    "    hidden_layer[hidden_layer < 0] = 0 # ReLU nonlinearity\n",
    "    yhat = np.dot(weights['W2'], hidden_layer)\n",
    "    up_probability = sigmoid(yhat)\n",
    "    return hidden_layer, up_probability # return probability of taking action 2, and hidden state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**[Backpropagation](http://neuralnetworksanddeeplearning.com/chap2.html):**\n",
    "  \n",
    "We want to minimize cost.\n",
    "* $z^l_j = \\sum_k w^l_{jk} a^{l-1}_k$ which vectorized is $z^{l} = w^l a^{l-1}$.\n",
    "> $z^l_j$ is pretty much $a^{l}_j$ without the activation function.\n",
    "* $\\delta^l_j = \\frac{\\partial C}{\\partial z^l_j}$ the error of neuron $j$ in layer $l$\n",
    "* $\\delta^L = \\nabla_a C \\odot \\sigma'(z^L)$\n",
    "> $\\nabla_a C$ measures how fast the cost function is changing with respect to the output activations while $\\sigma'(z^L)$ measures how fast the activation function is changing at $z^L$. For us, $\\nabla_a C$ is going to be $y - a^L$ and $\\sigma'(z^L)$ is going to be the discounted rewards we'll create (as part of our policy gradient) that will be explained in the next section. In code, we call this ```game_residuals```.\n",
    "\n",
    "![Equations of backpropagation by Michael Nielsen](http://neuralnetworksanddeeplearning.com/images/tikz21.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def backward(game_hidden_layer_values, game_residuals):\n",
    "    \"\"\" backward pass. (game_hidden_layer_values is array of intermediate hidden states) \"\"\"\n",
    "    dW2 = np.dot(game_hidden_layer_values.T, game_residuals).ravel()\n",
    "    dh = np.outer(game_residuals, weights['W2'])\n",
    "    dh[game_hidden_layer_values <= 0] = 0 # backpro prelu\n",
    "    dW1 = np.dot(dh.T, game_delta_frames)\n",
    "    return {'W1':dW1, 'W2':dW2}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We use RMSprop which is an unpublished, adaptive learning rate proposed by Geoff Hinton (one of the first researchers to demonstrate the use of generalized backpropagation algorithm).\n",
    "\n",
    "**Pitfalls of Stochastic Gradient Descent**  \n",
    "SGD is dependent on the given instance (or the batch of instances) of the present iteration. This results in:\n",
    "* unstable update steps per iteration\n",
    "* slower convergence\n",
    "* optimization more likely to get stuck in a poor local minima\n",
    "\n",
    "SGD is supposed to approximate each step by taking the average of all the mini batches, but there are times when a parameter might get a gradient of 0.0001 for one batch, but then the next batch (possibly an outlier batch) gives a gradient of -0.009. This effectively destroys all the previous gradient information.\n",
    "\n",
    "RMSrop instead keeps a running average of its recent gradient magnitudes and divides the next gradient by this average so that loosely gradient values are normalized (it essentially divides the learning rate by an exponentially decaying average of squared gradients)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "gradient_buffer = {key : np.zeros_like(value) for key, value in weights.items()} # update buffers that add up gradients over a batch\n",
    "rmsprop_cache = {key : np.zeros_like(value) for key, value in weights.items()} # rmsprop memory\n",
    "\n",
    "def update_weights():\n",
    "    for key, value in weights.items():\n",
    "        g = gradient_buffer[key] # gradient\n",
    "        rmsprop_cache[key] = backprop_rate * rmsprop_cache[key] + (1 - backprop_rate) * g**2\n",
    "        weights[key] += learning_rate * g / (np.sqrt(rmsprop_cache[key]) + 1e-5)\n",
    "        gradient_buffer[key] = np.zeros_like(value) # reset batch gradient buffer\n",
    "    return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Supervised Learning vs Reinforced Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"supervised.png\" alt=\"Supervised Learning\" style=\"\">\n",
    "\n",
    "<img src=\"rl.png\" alt=\"Reinforcement Learning\" style=\"\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The biggest difference between a typical supervised learning problem with neural networks and reinforcement learning with a neural network comes down to access to labels. In a supervised problem, you already have labels for each sample so that you can do backprogagation immediately. In a RL problem, you don't have immediate access to a sample's label and must sample an action and assume that action was the right one to take (sampling an action makes this process stochastic); eventually, you'll get a reward and use that to encourage or discourage actions you took through the typical backpropagation method with actions you took right before being rewarded being weighted more heavily than those actions you took further back. This last concept in combination with backpropagation is called a \"policy gradient\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Policy gradient\n",
    "\n",
    "For the algorithm to learn, you must ask how changing the output probability (i.e. the probability of pressing UP) affects the result of winning the round. We first have to assume that each action we take is the true action we should have taken, until we get a reward from the environment (this is a solution to the credit assignment problem).\n",
    "\n",
    "To calculate the cost of our actions, we can use a binary logistic regression classifier (since we're only trying to predict two things, UP or DOWN).  \n",
    "$P(y = 1 \\mid x; w, b) = \\frac{1}{1 + e^{-(w^Tx +b)}} = \\sigma (w^Tx + b)$  \n",
    "whose maximum likelihood estimator is $C_i = \\sum_j y_{ij} \\log(\\sigma(f_j)) + (1 - y_{ij}) \\log(1 - \\sigma(f_j))$ where $f$ is the function that gives us the activations of the final layer.\n",
    "\n",
    "Now to figure out how the change in our cost is affected by our predictions, we calculate:  \n",
    "$\\partial{C_i} / \\partial{f_j} = y_{ij} - \\sigma(f_j)$ which is nothing more than our residuals (i.e. our assumed true label minus the predicted label).  \n",
    "We can easily put this in code:\n",
    "```python\n",
    "assumed_true_label = 1 if action == 2 else 0\n",
    "residuals = assumed_true_label - up_probability\n",
    "```\n",
    "\n",
    "We later take these residuals and \"discount\" them so that actions taken earlier in the round are less important than those you just took. This is part of our policy gradient and key to the neural net's back-progpagation algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "discount_rate = 0.99 # discount factor for reward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def discount_rewards(rewards):\n",
    "    \"\"\" Actions you took 20 steps before the end result are less important to the overall\n",
    "    result than an action you took a step ago. This implements that logic by discounting\n",
    "    the reward on previous actions based on how long ago they were taken \"\"\"\n",
    "    discounted_rewards = np.zeros_like(rewards)\n",
    "    running_add = 0\n",
    "    # earlier frames are discounted a lot more than rewards for later frames\n",
    "    for i in list(reversed(range(0, rewards.size))):\n",
    "        if rewards[i] != 0:\n",
    "            running_add = 0 # reset the sum, since this was a game boundary (pong specific!)\n",
    "        running_add = running_add * discount_rate + rewards[i]\n",
    "        discounted_rewards[i] = running_add\n",
    "    return discounted_rewards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def choose_action(probability):\n",
    "    \"\"\" this is the stochastic part of RL and what makes this policy\n",
    "    neural network different from a supervised learning problem \"\"\"\n",
    "    random_value = np.random.uniform()\n",
    "    if random_value < probability:\n",
    "        # signifies up in openai gym\n",
    "        return 2\n",
    "    else:\n",
    "        # signifies down in openai gym\n",
    "        return 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Let's Play!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "env = gym.make(\"Pong-v0\")\n",
    "frame = env.reset()\n",
    "previously_processed_frame = None # used in computing the difference frame\n",
    "delta_frames,hidden_layer_values = [], []\n",
    "residuals, rewards = [], []\n",
    "running_reward = None\n",
    "reward_sum = 0\n",
    "game_number = 0\n",
    "\n",
    "while True:\n",
    "    env.render()\n",
    "\n",
    "    # preprocess the frame, set input to network to be difference image\n",
    "    delta_frame, previously_processed_frame = process_frame(frame, previously_processed_frame)\n",
    "\n",
    "    # forward the policy network and sample an action from the returned probability\n",
    "    hidden_layer, up_probability = forward(delta_frame)\n",
    "    action = choose_action(up_probability)\n",
    "\n",
    "    # record various intermediates (needed later for backprop)\n",
    "    delta_frames.append(delta_frame) #frame\n",
    "    hidden_layer_values.append(hidden_layer) # hidden state\n",
    "\n",
    "    # assume every action taken was the true action and\n",
    "    # calculate gradient\n",
    "    assumed_true_label = 1 if action == 2 else 0 # a \"fake label\"\n",
    "    residuals.append(assumed_true_label - up_probability) # grad that encourages the action that was taken to be taken (see http://cs231n.github.io/neural-networks-2/#losses if confused)\n",
    "\n",
    "    # step the environment and get new measurements\n",
    "    frame, reward, done, info = env.step(action)\n",
    "    reward_sum += reward\n",
    "\n",
    "    rewards.append(reward) # record reward (has to be done after we call step() to get reward for previous action)\n",
    "\n",
    "    if done: # an episode finished\n",
    "        game_number += 1\n",
    "\n",
    "        # stack together all inputs, hidden states, action gradients, and rewards for this episode\n",
    "        game_delta_frames = np.vstack(delta_frames)\n",
    "        game_hidden_layer_values = np.vstack(hidden_layer_values)\n",
    "        game_residuals = np.vstack(residuals)\n",
    "        game_rewards = np.vstack(rewards)\n",
    "\n",
    "        # reset array memory\n",
    "        delta_frames, hidden_layer_values = [], []\n",
    "        residuals, rewards = [],[]\n",
    "\n",
    "        # compute the discounted reward backwards through time\n",
    "        discounted_game_rewards = discount_rewards(game_rewards)\n",
    "        # standardize the rewards to be unit normal (helps control the gradient estimator variance)\n",
    "        discounted_game_rewards -= np.mean(discounted_game_rewards)\n",
    "        discounted_game_rewards /= np.std(discounted_game_rewards)\n",
    "        game_residuals *= discounted_game_rewards # modulate the gradient with advantage (PG magic happens right here.)\n",
    "\n",
    "        weight_gradients = backward(game_hidden_layer_values, game_residuals)\n",
    "        for key in weights: gradient_buffer[key] += weight_gradients[key] # accumulate gradient over batch\n",
    "\n",
    "        # perform rmsprop parameter update every games_to_play episodes\n",
    "        if game_number % games_to_play == 0:\n",
    "            print(\"updating weights\")\n",
    "            update_weights()\n",
    "\n",
    "        # track progress\n",
    "        running_reward = reward_sum if running_reward is None else running_reward * 0.99 + reward_sum * 0.01\n",
    "        print('game_number: {0}'.format(game_number))\n",
    "        print('resetting env. episode reward total was {0}. running mean: {1}'.format(reward_sum, running_reward))\n",
    "        if game_number % 100 == 0: pickle.dump(weights, open('nn_weights_demo.pkl', 'wb'))\n",
    "        reward_sum = 0\n",
    "        frame = env.reset() # reset env\n",
    "        previously_processed_frame = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "10,200 games were played, with the neural net updating a total of 1,020 times; the AI wins by an average of 4 points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"demo.gif\" alt=\"Pong Demo\" style=\"\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "* [RL Overview](https://karpathy.github.io/2016/05/31/rl/)\n",
    "* [Karpathy implementation](https://github.com/dhruvp/atari-pong/blob/master/me_pong.py)\n",
    "* [Karpathy implemenation derivative](https://medium.com/@dhruvp/how-to-write-a-neural-network-to-play-pong-from-scratch-956b57d4f6e0#.q7etxmxfc)\n",
    "* [RMSprop](http://sebastianruder.com/optimizing-gradient-descent/index.html#rmsprop)\n",
    "* [RMSprop vs SGD](http://www.erogol.com/comparison-sgd-vs-momentum-vs-rmsprop-vs-momentumrmsprop/)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}