P02-C04-softmax-regression-gluon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiclass logistic regression with ``gluon``\n",
    "\n",
    "Now that we've built a [logistic regression model from scratch](./P02-C03-softmax-regression-scratch.ipynb), let's make this more efficient with ``gluon``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import mxnet as mx\n",
    "from mxnet import nd, autograd\n",
    "from mxnet import gluon\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll also want to set the compute context for our modeling. Feel free to go ahead and change this to mx.gpu(0) if you're running on an appropriately endowed machine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ctx = mx.cpu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MNIST Dataset\n",
    "\n",
    "First, we'll grab the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mnist = mx.test_utils.get_mnist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Iterators\n",
    "\n",
    "And load up two data iterators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "train_data = mx.io.NDArrayIter(mnist[\"train_data\"], mnist[\"train_label\"], batch_size, shuffle=True)\n",
    "test_data = mx.io.NDArrayIter(mnist[\"test_data\"], mnist[\"test_label\"], batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're also going to want to load up an iterator with *test* data. After we train on the training dataset we're going to want to test our model on the test data. Otherwise, for all we know, our model could be doing something stupid (or treacherous?) like memorizing the training examples and regurgitating the labels on command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiclass Logistic Regression\n",
    "\n",
    "Now we're going to define our model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = gluon.nn.Sequential()\n",
    "with net.name_scope():\n",
    "    net.add(gluon.nn.Dense(10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter initialization\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net.collect_params().initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax Cross Entropy Loss\n",
    "\n",
    "Note, we didn't have to include the softmax layer because MXNet's has an efficient function that simultaneously computes the softmax activation and cross-entropy loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loss = gluon.loss.SoftmaxCrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer\n",
    "\n",
    "And let's instantiate an optimizer to make our updates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation Metric\n",
    "\n",
    "This time, let's simplify the evaluation code by relying on MXNet's built-in ``metric`` package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate_accuracy(data_iterator, net):\n",
    "    numerator = 0.\n",
    "    denominator = 0.\n",
    "    data_iterator.reset()\n",
    "    for i, batch in enumerate(data_iterator):\n",
    "        data = batch.data[0].as_in_context(ctx).reshape((-1,784))\n",
    "        label = batch.label[0].as_in_context(ctx)\n",
    "        label_one_hot = nd.one_hot(label, 10)\n",
    "        output = net(data)        \n",
    "        predictions = nd.argmax(output, axis=1)\n",
    "        numerator += nd.sum(predictions == label)\n",
    "        denominator += data.shape[0]\n",
    "    return (numerator / denominator).asscalar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we initialized our model randomly, and because roughly one tenth of all examples belong to each fo the ten classes, we should have an accuracy in the ball park of .10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.081707805"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_accuracy(test_data, net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0. Loss: 0.502687432975, Train_acc 0.901153, Test_acc 0.907842\n",
      "Epoch 1. Loss: 0.474829963345, Train_acc 0.910848, Test_acc 0.915406\n",
      "Epoch 2. Loss: 0.462082007241, Train_acc 0.915278, Test_acc 0.917197\n",
      "Epoch 3. Loss: 0.453585840656, Train_acc 0.917944, Test_acc 0.918491\n",
      "Epoch 4. Loss: 0.447037873108, Train_acc 0.919493, Test_acc 0.919885\n",
      "Epoch 5. Loss: 0.441640226039, Train_acc 0.920659, Test_acc 0.921377\n",
      "Epoch 6. Loss: 0.437038913038, Train_acc 0.921658, Test_acc 0.921377\n",
      "Epoch 7. Loss: 0.433043815375, Train_acc 0.922458, Test_acc 0.922174\n",
      "Epoch 8. Loss: 0.429535013337, Train_acc 0.923291, Test_acc 0.922273\n",
      "Epoch 9. Loss: 0.426428586984, Train_acc 0.924157, Test_acc 0.922771\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "moving_loss = 0.\n",
    "\n",
    "for e in range(epochs):\n",
    "    train_data.reset()\n",
    "    for i, batch in enumerate(train_data):\n",
    "        data = batch.data[0].as_in_context(ctx).reshape((-1,784))\n",
    "        label = batch.label[0].as_in_context(ctx)\n",
    "        with autograd.record():\n",
    "            output = net(data)\n",
    "            cross_entropy = loss(output, label)\n",
    "        cross_entropy.backward()\n",
    "        trainer.step(data.shape[0])\n",
    "        \n",
    "        ##########################\n",
    "        #  Keep a moving average of the losses\n",
    "        ##########################\n",
    "        if i == 0:\n",
    "            moving_loss = np.mean(cross_entropy.asnumpy()[0])\n",
    "        else:\n",
    "            moving_loss = .99 * moving_loss + .01 * np.mean(cross_entropy.asnumpy()[0])\n",
    "            \n",
    "    test_accuracy = evaluate_accuracy(test_data, net)\n",
    "    train_accuracy = evaluate_accuracy(train_data, net)\n",
    "    print(\"Epoch %s. Loss: %s, Train_acc %s, Test_acc %s\" % (e, moving_loss, train_accuracy, test_accuracy))    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Now let's take a look at how to implement modern neural networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "For whinges or inquiries, [open an issue on  GitHub.](https://github.com/zackchase/mxnet-the-straight-dope)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}