06_cnn.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#default_exp cnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#hide\n",
    "from nbdev.showdoc import *\n",
    "# all_flag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Networks\n",
    "> Summary: Convolutional Neural Networks, CNNs, ConvNets, Gradient Descent, Backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convolutional Neural Networks (CNNs / ConvNets)\n",
    "\n",
    "Convolutional neural networks as very similar to the ordinary [feed-forward neural networks](05_neural_network.ipynb). They differ in the sense that CNNs assume explicitly that the inputs are images, which enables us to encode specific properties in the architecture to recognize certain patterns in the images. The CNNs make use of *spatial* nature of the data. It means, CNNs perceive the objects similar to our perception of different objects in nature. For example, we recognize various objects by their shapes, size and colors. These objects are combinations of edges, corners, color patches, etc. CNNs can use a variety of detectors (such as edge detectors, corner detectors) to interpret images. These detectors are called **filters** or **kernels**. The mathematical operator that takes an image and a filter as input and produces a filtered output (e.g. edges, corners, etc. ) is called **convolution**.\n",
    "\n",
    "![](images/cnn_filter.jpg)\n",
    "*Learned features in a CNN. [[Image Source](https://medium.com/diaryofawannapreneur/deep-learning-for-computer-vision-for-the-average-person-861661d8aa61)]*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNNs Architecture\n",
    "\n",
    "Convolutional Neural Networks have a different architecture than regular Neural Networks. CNNs are organized in 3 dimensions (width, height and depth). Also,\n",
    "Unlike ordinary neural networks that each neuron in one layer is connected to all the neurons in the next layer, in a CNN, only a small number of the neurons in the current layer connects to neurons in the next layer.\n",
    "\n",
    "![](images/cnn_architecture.png)\n",
    "*Architecture of a CNN. [[Image Source](https://www.mathworks.com/videos/introduction-to-deep-learning-what-are-convolutional-neural-networks--1489512765771.html)]*\n",
    "\n",
    "ConvNets have three types of layers: **Convolutional Layer**, **Pooling Layer** and **Fully-Connected Layer**. By stacking these layers we can construct a convolutional neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional Layer\n",
    "Convolutional layer applies a convolution operator on the input data using a filter and produces an output that is called **feature map**. The purpose of the convolution operation is to extract the high-level features such as edges, from the input image. The first ConvLayer is captures the Low-Level features such as edges, color, orientation, etc. Adding more layers enables the architecture to adapt to the high-level features as well, giving us a network which has the wholesome understanding of images in the dataset.\n",
    "\n",
    "We execute a convolution by sliding the filter over the input. At every location, an element-wise matrix multiplication is performed and sums the result onto the feature map.\n",
    "\n",
    "![](images/cnn_convolution.gif)\n",
    "*Left: the filter slides over the input. Right: the result is summed and added to the feature map. [[Image Source](https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2)]*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The example above was a convolution operation shown in 2D using a 3x3 filter. But in reality these convolutions are performed in 3D because an image is represented as a 3D matrix with dimensions of width, height and depth, where depth corresponds to color channels (RGB). Therefore, a convolution filter covers the entire depth of its input so it must be 3D as well.\n",
    "\n",
    "![](images/cnn_convolution_2.png)\n",
    "*The filter of size 5x5x3 slides over the volume of input. [[Image Source](https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2)]*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We perform many convolutions on our input, where each convolution operation uses a different filter. This results in different feature maps. At the end, we stack all of these feature maps together and form the final output of the convolution layer.\n",
    "\n",
    "![](images/cnn_convolution_3.png)\n",
    "*Example of two filters (green and red) over the volume of input. [[Image Source](https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2)]*\n",
    "\n",
    "In order to make our output non-linear, we pass the result of the convolution operation through an activation function (usually ReLU). Thus, the values in the final feature maps are not actually the sums, but the ReLU function applied to them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stride and Padding\n",
    "\n",
    "**Stride** is the size of the step we move the convolution filter at each step. The default value of the stride is 1.\n",
    "\n",
    "![](images/cnn_stride1.gif)\n",
    "*Stride with value of 1. [[Image Source](https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2)]*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we increase the size of stride the feature map will get smaller. The figure below demonstrates a stride of 2.\n",
    "\n",
    "![](images/cnn_stride2.gif)\n",
    "*Stride with value of 2. [[Image Source](https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2)]*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the size of the feature map feature is reduced in dimensionality as compared to the input. If we want to prevent the feature map from shrinking, we apply **padding** to surround the input with zeros.\n",
    "\n",
    "![](images/cnn_padding1.gif)\n",
    "*Stride = 1 with padding = 1. [[Image Source](https://www.cntk.ai/pythondocs/CNTK_103D_MNIST_ConvolutionalNeuralNetwork.html)]*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pooling Layer\n",
    "\n",
    "After a convolution layer we usually perform *pooling* to reduce the dimensionality. This allows us to reduce the number of parameters, which both shortens the training time and prevents overfitting. Pooling layers downsample each feature map independently, reducing the width and height and keeping the depth intact. *max pooling* is the most common types of pooling, which takes the maximum value in each window. Pooling does not have any parameters. It just decreases the size of the feature map while at the same time keeping the important information (i.e. dominant features).\n",
    "\n",
    "![](images/cnn_maxpooling.png)\n",
    "*Max pooling takes the largest value. [[Image Source](https://www.cntk.ai/pythondocs/CNTK_103D_MNIST_ConvolutionalNeuralNetwork.html)]*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameters\n",
    "\n",
    "When using ConvNets, there are certain *hyperparameters* that we need to determine.\n",
    "\n",
    "1. Filter size (kernel size): 3x3 filter are very common, but 5x5 and 7x7 are also used depending on the application.\n",
    "2. Filter count: How many filters do we want to use. It’s a power of two anywhere between 32 and 1024. The more filters, the more powerful model. However, there is a possibility of overfitting due to large amount of parameters. Therefore, we usually start off with a small number of filters at the initial layers, and gradually increase the count as we go deeper into the network.\n",
    "3. Stride: The common stride value is 1\n",
    "4. Padding:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fully Connected Layer (FC Layer)\n",
    "We often have a couple of fully connected layers after convolution and pooling layers. Fully connected layers work as a classifier on top of these learned features. The last fully connected layer outputs a N dimensional vector where N is the number of classes. For example, for a digit classification CNN, N would be 10 since we have 10 digits.\n",
    " Please note that the output of both convolution and pooling layers are 3D volumes, but a fully connected layer only accepts a 1D vector of numbers. Therefore, we ***flatten*** the 3D volume, meaning we convert the 3D volume into 1D vector.\n",
    "\n",
    "![](images/cnn_fc.jpeg)\n",
    "*A CNN to classify handwritten digits. [[Image Source](https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53)]*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training ConvNets\n",
    "\n",
    "Training CNNs is the same as ordinary neural networks. We apply backpropagation with gradient descent. For reading about training neural networks please see [here](05_neural_network.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ConvNets Architectures\n",
    "This section is adopted from Stanford University course [here](http://cs231n.github.io/convolutional-networks/). Convolutional Networks are often made up of only three layer types: CONV, POOL (i.e. Max Pooling), FC. Therefore, the most common architecture pattern is as follows:\n",
    "\n",
    "`INPUT -> [[CONV -> RELU]*N -> POOL?]*M -> [FC -> RELU]*K -> FC`\n",
    "\n",
    "where the `*` indicates repetition, and the `POOL?` indicates an optional pooling layer. Moreover, `N >= 0` (and usually `N <= 3`), `M >= 0`, `K >= 0` (and usually `K < 3`).\n",
    "\n",
    "There are several architectures of CNNs available  that are very popular:\n",
    "\n",
    "- LeNet\n",
    "- AlexNet\n",
    "- ZF Net\n",
    "- GoogLeNet\n",
    "- VGGNet\n",
    "- ResNet\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "\n",
    "As a practice, I created a ConvNet to classify latex symbols. Particularly, I download the HASY data set of handwritten symbols from [here](https://github.com/MartinThoma/HASY). It includes 369 classes including Arabic numerals and Latin characters. I split the dataset into 80% train, 20% test and trained the CNN on training set. For training I used the Google colab utilizing GPU computations. Here's the [link](https://colab.research.google.com/drive/1bHlH4bGh5uUg4W20VGD7D-1bNwjLT7BX)\n",
    "to colab notebook. I got the accuracy of 81.75% on the test set. It definitely has room to be improved. The architecture of the CNN is as follows:\n",
    "\n",
    "```\n",
    "Model: \"sequential\"\n",
    "_________________________________________________________________\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "conv2d (Conv2D)              (None, 32, 32, 128)       1280      \n",
    "_________________________________________________________________\n",
    "conv2d_1 (Conv2D)            (None, 32, 32, 128)       147584    \n",
    "_________________________________________________________________\n",
    "max_pooling2d (MaxPooling2D) (None, 16, 16, 128)       0         \n",
    "_________________________________________________________________\n",
    "dropout (Dropout)            (None, 16, 16, 128)       0         \n",
    "_________________________________________________________________\n",
    "conv2d_2 (Conv2D)            (None, 16, 16, 128)       409728    \n",
    "_________________________________________________________________\n",
    "max_pooling2d_1 (MaxPooling2 (None, 8, 8, 128)         0         \n",
    "_________________________________________________________________\n",
    "flatten (Flatten)            (None, 8192)              0         \n",
    "_________________________________________________________________\n",
    "dense (Dense)                (None, 128)               1048704   \n",
    "_________________________________________________________________\n",
    "dropout_1 (Dropout)          (None, 128)               0         \n",
    "_________________________________________________________________\n",
    "dense_1 (Dense)              (None, 128)               16512     \n",
    "_________________________________________________________________\n",
    "dense_2 (Dense)              (None, 369)               47601     \n",
    "=================================================================\n",
    "Total params: 1,671,409\n",
    "Trainable params: 1,671,409\n",
    "Non-trainable params: 0\n",
    "```\n",
    "\n",
    "In order to make this project more interesting, I converted the python-keras model into a Tenserflowjs model, then developed a simple Web application using Javascript, loaded the model and used it for predicting latex symbol by drawing symbols in a canvas. Here's the [GitHub link](https://github.com/sci2lab/tensorflowjs/tree/master/latexRecognizer) for the Web app. Below is a snapshot of how it works:\n",
    "\n",
    "![](images/cnn_latexrecognizer.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Utilizing Colab GPU Capability\n",
    "\n",
    "The complete code is shown below. However, I strongly recommend to execute it while you have access to GPU such as in Google colab notebook, otherwise it will be very slow.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense, Activation, Flatten, Dropout\n",
    "from sklearn.model_selection import train_test_split\n",
    "from tqdm import tqdm\n",
    "from PIL import Image\n",
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def get_num_lines(file_name):\n",
    "    \"\"\" Counts the number of lines in the file. \"\"\"\n",
    "\n",
    "    f = open(file_name, 'r')\n",
    "    counter = 0;\n",
    "    for i in f:\n",
    "        counter += 1\n",
    "    return counter\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def load_symbols(file_name):\n",
    "    \"\"\" Reads the file having symbols and create two maps: `id2latex` and `latex2id`\n",
    "    to encode the symbols and retrieve them easily. \"\"\"\n",
    "\n",
    "    id2latex = dict()\n",
    "    latex2id = dict()\n",
    "    f = open(file_name,'r')\n",
    "    next(f)\n",
    "    id = 0;\n",
    "    for line in tqdm(f, total=get_num_lines(file_name)):\n",
    "      _,latex,_,_ = line.split(',')\n",
    "      if latex not in latex2id:\n",
    "          latex2id[latex] = id\n",
    "          id2latex[id] = latex\n",
    "          id += 1\n",
    "    return (id2latex,latex2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def load_data(label_file_name,latex2id):\n",
    "    \"\"\" Reads the data file and create and return `data` and `labels` lists. \"\"\"\n",
    "    data = []\n",
    "    labels = []\n",
    "    f = open(label_file_name,'r')\n",
    "    next(f)\n",
    "    for line in tqdm(f, total=get_num_lines(label_file_name)):\n",
    "      image_path,symbol_id,latex,_ = line.split(',')\n",
    "      img = Image.open(os.path.join(image_path)).convert('L')\n",
    "      img_array = np.asarray(img).astype('float32')\n",
    "      data.append(img_array)\n",
    "      labels.append(latex2id[latex])\n",
    "    return (data,labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create id2latex and latex2id maps\n",
    "symbol_file_name = 'symbols.csv'\n",
    "data_file_name   = 'hasy-data-labels.csv'\n",
    "id2latex,latex2id = load_symbols(symbol_file_name)\n",
    "data,labels = load_data(data_file_name,latex2id)\n",
    "\n",
    "# Randomly pick an example and display it\n",
    "sample = 83643\n",
    "img = Image.fromarray(data[sample])\n",
    "plt.imshow(img)\n",
    "print(id2latex[labels[sample]])\n",
    "\n",
    "# Split the data into train and test sets\n",
    "train_data,test_data,train_labels,test_labels = train_test_split(data,labels,test_size=0.2)\n",
    "\n",
    "# Normalizing train and test data\n",
    "normalized_train_data = np.asarray(train_data)/255.0\n",
    "normalized_test_data = np.asarray(test_data)/255.0\n",
    "\n",
    "# One-hot encoding of labels for train and test datasets\n",
    "encoded_train_labels = np.array(keras.utils.to_categorical(train_labels))\n",
    "encoded_test_labels = np.array(keras.utils.to_categorical(test_labels))\n",
    "\n",
    "# Reshaping train and test sets, i.e. changing from (32, 32) to (32, 32, 1)\n",
    "normalized_train_data = normalized_train_data.reshape(-1,32,32,1)\n",
    "normalized_test_data = normalized_test_data.reshape(-1,32,32,1)\n",
    "\n",
    "print('Input shape = {}'.format(normalized_train_data.shape[1:]))\n",
    "print('Number of classes =  {}'.format(encoded_train_labels.shape[1]))\n",
    "\n",
    "# Define intial variables\n",
    "input_features = normalized_train_data.shape[1]\n",
    "n_classes = encoded_train_labels.shape[1]\n",
    "batch_size = 128\n",
    "epochs = 15\n",
    "\n",
    "# Define the CNN model\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Conv2D(128, (3,3), activation='relu', padding='same', input_shape=(input_features,input_features,1)))\n",
    "model.add(Conv2D(128,(3,3), activation='relu', padding='same'))\n",
    "model.add(MaxPooling2D((2,2)))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Conv2D(128,(5,5), activation='relu', padding='same'))\n",
    "model.add(MaxPooling2D((2,2)))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dense(n_classes, activation='softmax'))\n",
    "\n",
    "model.summary()\n",
    "\n",
    "# Train the model\n",
    "model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])\n",
    "model.fit(normalized_train_data,encoded_train_labels, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(normalized_test_data,encoded_test_labels))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Train on 134586 samples, validate on 33647 samples\n",
    "Epoch 1/15\n",
    "134586/134586 [==============================] - 55s 412us/sample - loss: 2.9116 - accuracy: 0.3689 - val_loss: 1.1192 - val_accuracy: 0.6942\n",
    "Epoch 2/15\n",
    "134586/134586 [==============================] - 48s 357us/sample - loss: 1.6461 - accuracy: 0.5727 - val_loss: 0.8604 - val_accuracy: 0.7529\n",
    "Epoch 3/15\n",
    "134586/134586 [==============================] - 48s 356us/sample - loss: 1.3399 - accuracy: 0.6383 - val_loss: 0.7842 - val_accuracy: 0.7749\n",
    "Epoch 4/15\n",
    "134586/134586 [==============================] - 48s 356us/sample - loss: 1.1756 - accuracy: 0.6739 - val_loss: 0.7115 - val_accuracy: 0.7893\n",
    "Epoch 5/15\n",
    "134586/134586 [==============================] - 48s 354us/sample - loss: 1.0635 - accuracy: 0.7013 - val_loss: 0.7017 - val_accuracy: 0.7889\n",
    "Epoch 6/15\n",
    "134586/134586 [==============================] - 48s 355us/sample - loss: 0.9863 - accuracy: 0.7164 - val_loss: 0.6497 - val_accuracy: 0.8052\n",
    "Epoch 7/15\n",
    "134586/134586 [==============================] - 48s 354us/sample - loss: 0.9254 - accuracy: 0.7306 - val_loss: 0.6536 - val_accuracy: 0.8029\n",
    "Epoch 8/15\n",
    "134586/134586 [==============================] - 48s 356us/sample - loss: 0.8730 - accuracy: 0.7439 - val_loss: 0.6280 - val_accuracy: 0.8069\n",
    "Epoch 9/15\n",
    "134586/134586 [==============================] - 48s 354us/sample - loss: 0.8305 - accuracy: 0.7530 - val_loss: 0.6123 - val_accuracy: 0.8149\n",
    "Epoch 10/15\n",
    "134586/134586 [==============================] - 48s 356us/sample - loss: 0.7942 - accuracy: 0.7614 - val_loss: 0.6133 - val_accuracy: 0.8117\n",
    "Epoch 11/15\n",
    "134586/134586 [==============================] - 48s 354us/sample - loss: 0.7660 - accuracy: 0.7686 - val_loss: 0.5940 - val_accuracy: 0.8172\n",
    "Epoch 12/15\n",
    "134586/134586 [==============================] - 48s 355us/sample - loss: 0.7425 - accuracy: 0.7735 - val_loss: 0.5928 - val_accuracy: 0.8185\n",
    "Epoch 13/15\n",
    "134586/134586 [==============================] - 48s 355us/sample - loss: 0.7229 - accuracy: 0.7775 - val_loss: 0.5971 - val_accuracy: 0.8167\n",
    "Epoch 14/15\n",
    "134586/134586 [==============================] - 48s 355us/sample - loss: 0.7039 - accuracy: 0.7822 - val_loss: 0.5893 - val_accuracy: 0.8198\n",
    "Epoch 15/15\n",
    "134586/134586 [==============================] - 48s 355us/sample - loss: 0.6867 - accuracy: 0.7867 - val_loss: 0.5890 - val_accuracy: 0.8175\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "score = model.evaluate(normalized_test_data, encoded_test_labels, verbose=0)\n",
    "print('Test loss: {:.3f}'.format(score[0]))\n",
    "print('Test accuracy: {:.2f}%'.format(score[1] * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Test loss: 0.589\n",
    "Test accuracy: 81.75%\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extra Resources\n",
    "[1] [Stanford course on Convolutional Neural networks](http://cs231n.github.io/convolutional-networks/)\n",
    "\n",
    "[2] [A Beginner's Guide To Understanding Convolutional Neural Networks](https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional-Neural-Networks/)\n",
    "\n",
    "[3] [Convolutional Neural Networks](https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}