Alogrithems/DL0101EN-3-2-Classification-with-Keras-py-v1.0.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.5"
    },
    "colab": {
      "name": "DL0101EN-3-2-Classification-with-Keras-py-v1.0.ipynb",
      "provenance": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "gdo8pshHCykX",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://cognitiveclass.ai\"><img src = \"https://ibm.box.com/shared/static/9gegpsmnsoo25ikkbl4qzlvlyjbgxs5x.png\" width = 400> </a>\n",
        "\n",
        "<h1 align=center><font size = 5>Classification Models with Keras</font></h1>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "RU6hqZsPCykY",
        "colab_type": "text"
      },
      "source": [
        "## Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "BF6NC29ECykZ",
        "colab_type": "text"
      },
      "source": [
        "In this lab, we will learn how to use the Keras library to build models for classificaiton problems. We will use the popular MNIST dataset, a dataset of images, for a change. \n",
        "\n",
        "The <strong>MNIST database</strong>, short for Modified National Institute of Standards and Technology database, is a large database of handwritten digits that is commonly used for training various image processing systems.he database is also widely used for training and testing in the field of machine learning.\n",
        "    \n",
        "The MNIST database contains 60,000 training images and 10,000 testing images of digits written by high school students and employees of the United States Census Bureau.\n",
        "\n",
        "Also, this way, will get to compare how conventional neural networks compare to convolutional neural networks, that we will build in the next module."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "yFYJDIBCCyka",
        "colab_type": "text"
      },
      "source": [
        "## Table of Contents\n",
        "\n",
        "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
        "\n",
        "<font size = 3>\n",
        "\n",
        "1. <a href=\"#item2\">Import Keras and Packages</a>      \n",
        "2. <a href=\"#item3\">Build a Neural Network</a>     \n",
        "3. <a href=\"#item4\">Train and Test the Network</a>     \n",
        "</font>\n",
        "</div>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "4UZx49vsCykc",
        "colab_type": "text"
      },
      "source": [
        "## Import Keras and Packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "sX4xtY6rCykd",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "1f797c15-6a02-480e-84c5-715c50212c07"
      },
      "source": [
        "import keras\n",
        "\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from keras.utils import to_categorical"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "wIRHzwlOCykh",
        "colab_type": "text"
      },
      "source": [
        "Since we are dealing we images, let's also import the Matplotlib scripting layer in order to view the images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "lvBMSS10Cyki",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "LopeWatjCykl",
        "colab_type": "text"
      },
      "source": [
        "The Keras library conveniently includes the MNIST dataset as part of its API. You can check other datasets within the Keras library [here](https://keras.io/datasets/). \n",
        "\n",
        "So, let's load the MNIST dataset from the Keras library. The dataset is readily divided into a training set and a test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "itlutUjoCykm",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "42551265-8a99-4484-b640-7c2249bb4a17"
      },
      "source": [
        "# import the data\n",
        "from keras.datasets import mnist\n",
        "\n",
        "# read the data\n",
        "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 1s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "1w22fX0pCykq",
        "colab_type": "text"
      },
      "source": [
        "Let's confirm the number of images in each set. According to the dataset's documentation, we should have 60000 images in X_train and 10000 images in the X_test."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "JK1MDnhSCykr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "X_train.shape"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "QIagO_0oCykv",
        "colab_type": "text"
      },
      "source": [
        "The first number in the output tuple is the number of images, and the other two numbers are the size of the images in datset. So, each image is 28 pixels by 28 pixels."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "-DaxWSybCykw",
        "colab_type": "text"
      },
      "source": [
        "Let's visualize the first image in the training set using Matplotlib's scripting layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "UlkrTSOcCykw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plt.imshow(X_train[0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "VCjA4KJzCyk2",
        "colab_type": "text"
      },
      "source": [
        "With conventional neural networks, we cannot feed in the image as input as is. So we need to flatten the images into one-dimensional vectors, each of size 1 x (28 x 28) = 1 x 784."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "IvLLkXOBCyk2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# flatten images into one-dimensional vector\n",
        "\n",
        "num_pixels = X_train.shape[1] * X_train.shape[2] # find size of one-dimensional vector\n",
        "\n",
        "X_train = X_train.reshape(X_train.shape[0], num_pixels).astype('float32') # flatten training images\n",
        "X_test = X_test.reshape(X_test.shape[0], num_pixels).astype('float32') # flatten test images"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "e2XeY4__Cyk7",
        "colab_type": "text"
      },
      "source": [
        "Since pixel values can range from 0 to 255, let's normalize the vectors to be between 0 and 1."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "7Ppkx7AkCyk7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# normalize inputs from 0-255 to 0-1\n",
        "X_train = X_train / 255\n",
        "X_test = X_test / 255"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "R4QlArIUCyk-",
        "colab_type": "text"
      },
      "source": [
        "Finally, before we start building our model, remember that for classification we need to divide our target variable into categories. We use the to_categorical function from the Keras Utilities package."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "GVYPr1wLCyk_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8ba09bd7-2d2a-40b4-8cea-b32f792bf543"
      },
      "source": [
        "# one hot encode outputs\n",
        "y_train = to_categorical(y_train)\n",
        "y_test = to_categorical(y_test)\n",
        "\n",
        "num_classes = y_test.shape[1]\n",
        "print(num_classes)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "10\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "ntCIFmWoCylG",
        "colab_type": "text"
      },
      "source": [
        "## Build a Neural Network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "5D7AwT7ECylH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# define classification model\n",
        "def classification_model():\n",
        "    # create model\n",
        "    model = Sequential()\n",
        "    model.add(Dense(num_pixels, activation='relu', input_shape=(num_pixels,)))\n",
        "    model.add(Dense(100, activation='relu'))\n",
        "    model.add(Dense(num_classes, activation='softmax'))\n",
        "    \n",
        "    \n",
        "    # compile model\n",
        "    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
        "    return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "cDNgqvNdCylL",
        "colab_type": "text"
      },
      "source": [
        "## Train and Test the Network"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "0Z_xgRW3CylM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        },
        "outputId": "13d2a761-78d9-4fac-bc48-35fa160a686b"
      },
      "source": [
        "# build the model\n",
        "model = classification_model()\n",
        "\n",
        "# fit the model\n",
        "model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, verbose=2)\n",
        "\n",
        "# evaluate the model\n",
        "scores = model.evaluate(X_test, y_test, verbose=0)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 60000 samples, validate on 10000 samples\n",
            "Epoch 1/10\n",
            " - 20s - loss: 0.1845 - accuracy: 0.9442 - val_loss: 0.1039 - val_accuracy: 0.9682\n",
            "Epoch 2/10\n",
            " - 22s - loss: 0.0789 - accuracy: 0.9751 - val_loss: 0.1115 - val_accuracy: 0.9672\n",
            "Epoch 3/10\n",
            " - 21s - loss: 0.0528 - accuracy: 0.9834 - val_loss: 0.0808 - val_accuracy: 0.9757\n",
            "Epoch 4/10\n",
            " - 20s - loss: 0.0400 - accuracy: 0.9870 - val_loss: 0.0797 - val_accuracy: 0.9784\n",
            "Epoch 5/10\n",
            " - 20s - loss: 0.0312 - accuracy: 0.9900 - val_loss: 0.0778 - val_accuracy: 0.9791\n",
            "Epoch 6/10\n",
            " - 20s - loss: 0.0258 - accuracy: 0.9912 - val_loss: 0.0787 - val_accuracy: 0.9789\n",
            "Epoch 7/10\n",
            " - 20s - loss: 0.0213 - accuracy: 0.9932 - val_loss: 0.0753 - val_accuracy: 0.9808\n",
            "Epoch 8/10\n",
            " - 20s - loss: 0.0214 - accuracy: 0.9931 - val_loss: 0.0820 - val_accuracy: 0.9797\n",
            "Epoch 9/10\n",
            " - 20s - loss: 0.0159 - accuracy: 0.9947 - val_loss: 0.1161 - val_accuracy: 0.9768\n",
            "Epoch 10/10\n",
            " - 21s - loss: 0.0171 - accuracy: 0.9945 - val_loss: 0.1127 - val_accuracy: 0.9767\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "-cNgFr-zCylO",
        "colab_type": "text"
      },
      "source": [
        "Let's print the accuracy and the corresponding error."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "P-vEKshKCylP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print('Accuracy: {}% \\n Error: {}'.format(scores[1], 1 - scores[1]))        "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "i61-r3ZBCylT",
        "colab_type": "text"
      },
      "source": [
        "Just running 10 epochs could actually take over 2 minutes. But enjoy the results as they are getting generated."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "Ky41D6T-CylU",
        "colab_type": "text"
      },
      "source": [
        "Sometimes, you cannot afford to retrain your model everytime you want to use it, especially if you are limited on computational resources and training your model can take a long time. Therefore, with the Keras library, you can save your model after training. To do that, we use the save method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "_-zUTLtCCylV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.save('classification_model.h5')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "Qj9LGgGdCylY",
        "colab_type": "text"
      },
      "source": [
        "Since our model contains multidimensional arrays of data, then models are usually saved as .h5 files."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "hKHF_I01CylY",
        "colab_type": "text"
      },
      "source": [
        "When you are ready to use your model again, you use the load_model function from <strong>keras.models</strong>."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "9ttSbvD6CylZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.models import load_model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "PedruOuVCylc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pretrained_model = load_model('classification_model.h5')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "tT7I7y4ICylf",
        "colab_type": "text"
      },
      "source": [
        "### Thank you for completing this lab!\n",
        "\n",
        "This notebook was created by [Alex Aklson](https://www.linkedin.com/in/aklson/). I hope you found this lab interesting and educational. Feel free to contact me if you have any questions!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "GBHdhDgyCylg",
        "colab_type": "text"
      },
      "source": [
        "This notebook is part of a course on **edX** called *Deep Learning Fundamentals with Keras*. If you accessed this notebook outside the course, you can take this course online by clicking [here](http://cocl.us/DL0101EN_edX_Week3_LAB2)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "g46UxEnsCylg",
        "colab_type": "text"
      },
      "source": [
        "<hr>\n",
        "\n",
        "Copyright &copy; 2018 [IBM Developer Skills Network](https://cognitiveclass.ai/?utm_source=bducopyrightlink&utm_medium=dswb&utm_campaign=bdu). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/)."
      ]
    }
  ]
}