LeNet5_PyTorch.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Build the architecture of LeNet5 from Scratch\n",
        "\n",
        "# [Link to my Youtube Video Explaining this whole Notebook](https://www.youtube.com/watch?v=Uq5sQUoLXpA&list=PLxqBkZuBynVRyOJs4RWmB_fKlOVe5S8CR&index=9)\n",
        "\n",
        "[![Imgur](https://imgur.com/yEmSfK0.png)](https://www.youtube.com/watch?v=Uq5sQUoLXpA&list=PLxqBkZuBynVRyOJs4RWmB_fKlOVe5S8CR&index=9)\n",
        "\n",
        "There are two main steps after that. \n",
        "\n",
        "First is initializing the layers that we are going to use in our CNN inside __init__ , and \n",
        "\n",
        "Then the other is to define the sequence in which those layers will process the image. This is defined inside the forward function."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Imgur](https://imgur.com/yrIrojL.png)\n",
        "\n",
        "The above diagram shows a description of the LeNet-5 architecture as shown in the original document.\n",
        "\n",
        " \n",
        "**Layer 1**- The first layer is the input layer; It is generally not considered a layer of the network as nothing is learned on that layer. The input layer supports 32x32, and these are the dimensions of the images that will be passed to the next layer.\n",
        "\n",
        "The grayscale images used in the research paper had their pixel values normalized from 0 to 255, to values between -0.1 and 1.175. The reason for normalization is to ensure that the batch of images have a mean of 0 and a standard deviation of 1, the benefits of this is seen in the reduction in the amount of training time. In the image classification with LeNet-5 example below, we’ll be normalizing the pixel values of the images to take on values between 0 to 1.\n",
        "\n",
        " \n",
        "**Layer 2**- Layer C1 is a convolution layer with six 5 × 5 convolution kernels, and the feature allocation size is 28 × 28, whereby input image information can be avoided.\n",
        "\n",
        " \n",
        "**Layer 3**- Layer S2 is the undersampling / grouping layer which generates 6 function graphs of length 14x14. Each cell in every function map is attached to 2x2 neighborhoods at the corresponding function map in C1. \n",
        " \n",
        "**Layer 4** - C3 convolution layer encompass sixteen 5x5 convolution kernels The input of the primary six function maps C3 is every continuous subset of the 3 function maps in S2, the access of the following six function maps comes from the access of the 4 continuous subsets and the input for the following 3 function maps is crafted from the 4 discontinuous subsets.Finally, the input for the very last function diagram comes from all the S2 function diagrams. \n",
        " \n",
        "**Layer 5**- Layer S4 is just like S2 with a length of 2x2 and an output of sixteen 5x5 function graphics. \n",
        "\n",
        "**Layer 6**- Layer C5 is a convolution layer with one hundred twenty convolution cores of length 5x5. Each cell is attached to the 5x5 neighborhoods along sixteen S4 function charts. Since the function chart length of S4 is likewise 5x5, the output length of C5 is 1 * 1, so S4 and C5 are absolutely linked.\n",
        "\n",
        "It is referred to as a convolutional layer in preference to a completely linked layer due to the fact if the input of LeNet-5 becomes large and its shape stays unchanged, then its output length is bigger than 1x1, i.e. now no longer a completely linked layer.\n",
        " \n",
        "\n",
        "**Layer 7-** The F6 layer is connected to C5 and 84 feature charts are generated. In the grayscale images used in the research, the pixel values ​​from 0 to 255 were normalized to values ​​between -0.1 and 1,175 The reason for normalization is to make sure the image stack has a mean of 0 and a standard deviation of 1.\n",
        "\n",
        "The advantages of this are in the reduction of the training time. In the following example we will normalize the pixel values ​​of the images to take values ​​between 0 and 1."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Vx23hoSrw87A",
        "outputId": "cfd18dde-d1a9-48ca-8107-a93b87d70c9f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "device(type='cuda')"
            ]
          },
          "execution_count": 27,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torchvision\n",
        "import torchvision.transforms as transforms\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "# Device will determine whether to run the training on GPU or CPU.\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "# device = 'cpu'\n",
        "device"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SBNQhORMxEpf",
        "outputId": "ab8eca2e-2d95-4535-8de0-c465fedf4577"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Wed Feb 16 15:15:37 2022       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   48C    P0    35W / 250W |   1065MiB / 16280MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ],
      "source": [
        "!nvidia-smi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "Dt8ZQ7JeGcVL"
      },
      "outputs": [],
      "source": [
        "batch_size = 64\n",
        "num_classes = 10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "![Imgur](https://imgur.com/yrIrojL.png)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "#Defining the convolutional neural network\n",
        "class LeNet5(nn.Module):\n",
        "    def __init__(self, num_classes):\n",
        "        super(LeNet5, self).__init__()\n",
        "        self.layer1 = nn.Sequential(\n",
        "            nn.Conv2d(1, 6, kernel_size=5, stride=1, padding=0),\n",
        "            # Above is Layer-1 - The input for LeNet-5 is a 32×32 grayscale image which passes through the first convolutional layer with 6 feature maps or filters having size 5×5 and a stride of one. The image dimensions changes from 32x32x1 to 28x28x6.\n",
        "            \n",
        "            nn.BatchNorm2d(6),\n",
        "            nn.ReLU(),\n",
        "            # Layer-2 - Then average pooling layer or sub-sampling layer with a filter size 2×2 and a stride of two. The resulting image dimensions will be reduced to 14x14x6.          \n",
        "            nn.MaxPool2d(kernel_size = 2, stride = 2)\n",
        "        )      \n",
        "        \n",
        "        self.layer2 = nn.Sequential(\n",
        "            nn.Conv2d(6, 16, kernel_size=5, stride=1, padding=0),\n",
        "            # Layer-3 Next, there is a second convolutional layer with 16 feature maps having size 5×5 and a stride of 1. In this layer, only 10 out of 16 feature maps are connected to 6 feature maps of the previous layer\n",
        "            nn.BatchNorm2d(16),\n",
        "            nn.ReLU(),\n",
        "            # Layer-4 The fourth layer (S4) is again an average pooling layer with filter size 2×2 and a stride of 2. This layer is the same as the second layer (S2) except it has 16 feature maps so the output will be reduced to 5x5x16.\n",
        "            nn.MaxPool2d(kernel_size = 2, stride = 2)\n",
        "        )\n",
        "        \n",
        "        self.fc = nn.Linear(400, 120)\n",
        "        ''' Layer-5 The fifth layer (C5) is a fully connected convolutional layer with 120 feature maps each of size 1×1. Each of the 120 units in C5 is connected to all the 400 nodes (5x5x16) in the fourth layer S4. '''\n",
        "        self.relu = nn.ReLU()\n",
        "        '''Layer-6 The sixth layer is a fully connected layer (F6) with 84 units. '''\n",
        "        self.fc1 = nn.Linear(120, 84)\n",
        "        self.relu1 = nn.ReLU()\n",
        "        ''' Layer-7 - Finally, a fully connected layer ŷ with 10 possible values corresponding to the digits from 0 to 9. Since the MNIST data has 10 classes for each of the 10 numerical digits. '''\n",
        "        self.fc2 = nn.Linear(84, num_classes)\n",
        "        \n",
        "    def forward(self, x):\n",
        "        output = self.layer1(x)\n",
        "        output = self.layer2(output)\n",
        "        print('output after layer2', output.size()) # torch.Size([32, 16, 5, 5]\n",
        "        # inside forward method image dimension are : [batch_size, channels, height, width]\n",
        "        output = output.reshape(output.size(0), -1)\n",
        "        # print('output after resize', output.size()) # torch.Size([32, 400])\n",
        "        output = self.fc(output)\n",
        "        output = self.relu(output)\n",
        "        output = self.fc1(output)\n",
        "        output = self.relu1(output)\n",
        "        output = self.fc2(output)\n",
        "        return output"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Reason to reshape after `output = self.layer2(output)`\n",
        "\n",
        "In PyTorch, images are represented as [channels, height, width], so a color image would be [3, 256, 256].\n",
        "\n",
        "During the training you will get batches of images, so your shape in the forward method will get an additional batch dimension at dim0: [batch_size, channels, height, width].\n",
        "\n",
        "So before applying the fully connected layer with output-shape of 400, I have to convert the above 4-D Tensor to a 2-D Tensor. \n",
        "And I definitely have to keep the first dimension, which is the batch-size, hence to the `.reshape()` function I am passing the unchanged first dimension of `output.size(0)`\n",
        "\n",
        "But for the second dimension, I am letting PyTorch to decide based on the Matrix calculation. i.e. the second dimension will be inferred given the first dimension.\n",
        "\n",
        "\n",
        "-----------------\n",
        "\n",
        "## where is softmax in above ? \n",
        "\n",
        "If you thought that the last layer in a Neural Network should be some sort of activation function like sigmoid() or softmax(), and we dont see that happening here in the above function.\n",
        "\n",
        "So, where is softmax? And its right here:\n",
        "\n",
        "```py\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "```\n",
        "\n",
        "Inside `nn.CrossEntropyLoss()` function is handled the softmax computation which, of course, works with the raw output of your last layer\n",
        "\n",
        "---\n",
        "\n",
        "## Setting Hyperparameters\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XHOaF28qw87C"
      },
      "source": [
        "### Loading the Dataset\n",
        "Using torchvision , we will load the dataset as this will allow us to perform any pre-processing steps easily."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "kqUweCDjGe2R"
      },
      "outputs": [],
      "source": [
        "class LeNet5(nn.Module):\n",
        "    def __init__(self,num_classes):\n",
        "        super(LeNet5,self).__init__()\n",
        "        self.layer1 = nn.Sequential(\n",
        "            nn.Conv2d(in_channels=1, out_channels=6, kernel_size=(5,5), stride=(1,1), padding=(0,0)), #Layer 1\n",
        "            nn.BatchNorm2d(6),\n",
        "            nn.ReLU(),\n",
        "            nn.MaxPool2d(kernel_size=(2, 2), stride=(2)) #Layer-2\n",
        "        )                \n",
        "        \n",
        "        self.layer2 = nn.Sequential(\n",
        "            nn.Conv2d(in_channels=6, out_channels=16, kernel_size=(5,5), stride=(1,1), padding=(0,0)), #Layer 3\n",
        "            nn.BatchNorm2d(16),\n",
        "            nn.ReLU(),\n",
        "            nn.MaxPool2d(kernel_size=(2, 2), stride=(2,2)) #Layer 4\n",
        "        )\n",
        "        self.fc = nn.Linear(400, 120) # Layer 5\n",
        "        self.relu = nn.ReLU()\n",
        "        self.fc1 = nn.Linear(120, 84) # Layer 6\n",
        "        self.relu1 = nn.ReLU()\n",
        "        self.fc2 = nn.Linear(84, num_classes) # Final Layer\n",
        "        \n",
        "    def forward(self, x):\n",
        "        output = self.layer1(x)\n",
        "        output = self.layer2(output)\n",
        "        # print('output after layer2', output.size()) # torch.Size([32, 16, 5, 5]\n",
        "        output = output.reshape(output.size(0), -1) # See note below for this line\n",
        "        # print('output after resize', output.size()) # torch.Size([32, 400])\n",
        "        output = self.fc(output)\n",
        "        output = self.relu(output)\n",
        "        output = self.fc1(output)\n",
        "        output = self.relu1(output)\n",
        "        output = self.fc2(output)\n",
        "        return output\n",
        "        "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "hg3ntm7uw87E"
      },
      "outputs": [],
      "source": [
        "# Define relevant variables for the ML task\n",
        "\n",
        "''' Keeping num_classes at 10, as this will be the output shape from the final Layer of the LeNet5 Neural Network model, because, the output layer will have 10 output neurons, since the MNIST data has 10 classes for each of the 10 numerical digits. '''\n",
        "\n",
        "train_dataset = torchvision.datasets.MNIST(root = './data',\n",
        "                                           train = True,\n",
        "                                           transform = transforms.Compose([\n",
        "                                                  transforms.Resize((32,32)),\n",
        "                                                  transforms.ToTensor(),\n",
        "                                                  transforms.Normalize(mean = (0.1307,), std = (0.3081,))]),\n",
        "                                           download = True)\n",
        "\n",
        "\n",
        "test_dataset = torchvision.datasets.MNIST(root = './data',\n",
        "                                          train = False,\n",
        "                                          transform = transforms.Compose([\n",
        "                                                  transforms.Resize((32,32)),\n",
        "                                                  transforms.ToTensor(),\n",
        "                                                  transforms.Normalize(mean = (0.1325,), std = (0.3105,))]),\n",
        "                                          download=True)\n",
        "\n",
        "\n",
        "train_dataloader = torch.utils.data.DataLoader(dataset = train_dataset,\n",
        "                                           batch_size = batch_size,\n",
        "                                           shuffle = True)\n",
        "\n",
        "\n",
        "test_dataloader = torch.utils.data.DataLoader(dataset = test_dataset,\n",
        "                                           batch_size = batch_size,\n",
        "                                           shuffle = True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PJSNQLNAw87G"
      },
      "source": [
        "## Build the architecture of LeNet5 from Scratch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "QuJAdibTw87G"
      },
      "outputs": [],
      "source": [
        "learning_rate = 0.001\n",
        "\n",
        "model = LeNet5(num_classes).to(device)\n",
        "\n",
        "#Setting the loss function\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "#Setting the optimizer with the model parameters and learning rate\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
        "\n",
        "#this is defined to print how many steps are remaining when training\n",
        "total_step = len(train_dataloader)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cLjcifcvGj1H",
        "outputId": "22c12c91-675b-4536-c44b-267fa911dcf1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Length of train_dataloader  938\n",
            "Length of test_dataloader  157\n"
          ]
        }
      ],
      "source": [
        "print('Length of train_dataloader ', len(train_dataloader))\n",
        "print('Length of test_dataloader ', len(test_dataloader))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZbZJsUIvw87H"
      },
      "source": [
        "## Setting Hyperparameters\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "Q3f-rT9_w87J"
      },
      "outputs": [],
      "source": [
        "def train(model, criterion, optimizer, num_epochs=10):\n",
        "    total_training_loss = []\n",
        "    total_step = len(train_dataloader)\n",
        "    \n",
        "    for epoch in range(num_epochs):\n",
        "        running_loss = 0.0\n",
        "        \n",
        "        for i, (images, labels) in enumerate(train_dataloader):  \n",
        "            images = images.to(device)\n",
        "            labels = labels.to(device)\n",
        "            \n",
        "            #Forward pass\n",
        "            outputs = model(images)\n",
        "            loss = criterion(outputs, labels)\n",
        "                \n",
        "            # Backward and optimize\n",
        "            optimizer.zero_grad() # Clear the past gradient by set the gradients to zero before every update\n",
        "            loss.backward() #  calculate the new gradients\n",
        "            # print('images.size ', images.size(0))\n",
        "            '''  \"\"\" Update the running loss \n",
        "            we need to use, loss.item() instead of loss alone in running_loss calculation and averaging. \n",
        "            Because loss gives you a grad_function, not a float value. \n",
        "            The item() method extracts the loss’s value as a Python float.\n",
        "            \"\"\" '''\n",
        "            running_loss += loss.item() * images.size(0)\n",
        "            optimizer.step() # we update the weights \n",
        "            \n",
        "                \n",
        "            if (i+1) % 400 == 0:\n",
        "                print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}' \n",
        "                            .format(epoch+1, num_epochs, i+1, total_step, loss.item()))\n",
        "        epoch_loss = running_loss / len(train_dataloader)\n",
        "        total_training_loss.append(epoch_loss)\n",
        "        \"\"\" To summarize, in the above code what I did is\n",
        "      \n",
        "        1. => multiply each average batch loss with batch-length. \n",
        "        The batch-length is inputs.size(0) which gives the number total images in each batch. \n",
        "        Essentially I am un-averaging the Batch-Loss\n",
        "\n",
        "        Do this all the batches inside the batch-running loop.\n",
        "        And then after that loop, i.e. outside the batch-loop and coming back to my epoch-loop\n",
        "        \n",
        "        2. => Divide this accumulated un-averaged Batch-loss from all batches, \n",
        "        by the number of samples (len(train_dataloader)) to get the exact train loss average for the epoch \"\"\"\n",
        "    return total_training_loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KHSSd1-gC9n4",
        "outputId": "49b9194c-9e42-471e-a3e4-f2b449d018bc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch [1/30], Step [400/938], Loss: 0.0270\n",
            "Epoch [1/30], Step [800/938], Loss: 0.1064\n",
            "Epoch [2/30], Step [400/938], Loss: 0.0322\n",
            "Epoch [2/30], Step [800/938], Loss: 0.0453\n",
            "Epoch [3/30], Step [400/938], Loss: 0.0066\n",
            "Epoch [3/30], Step [800/938], Loss: 0.0386\n",
            "Epoch [4/30], Step [400/938], Loss: 0.0346\n",
            "Epoch [4/30], Step [800/938], Loss: 0.0306\n",
            "Epoch [5/30], Step [400/938], Loss: 0.0016\n",
            "Epoch [5/30], Step [800/938], Loss: 0.0295\n",
            "Epoch [6/30], Step [400/938], Loss: 0.0009\n",
            "Epoch [6/30], Step [800/938], Loss: 0.0145\n",
            "Epoch [7/30], Step [400/938], Loss: 0.0020\n",
            "Epoch [7/30], Step [800/938], Loss: 0.0363\n",
            "Epoch [8/30], Step [400/938], Loss: 0.0061\n",
            "Epoch [8/30], Step [800/938], Loss: 0.0026\n",
            "Epoch [9/30], Step [400/938], Loss: 0.0335\n",
            "Epoch [9/30], Step [800/938], Loss: 0.0120\n",
            "Epoch [10/30], Step [400/938], Loss: 0.0395\n",
            "Epoch [10/30], Step [800/938], Loss: 0.0007\n",
            "Epoch [11/30], Step [400/938], Loss: 0.0007\n",
            "Epoch [11/30], Step [800/938], Loss: 0.0060\n",
            "Epoch [12/30], Step [400/938], Loss: 0.0012\n",
            "Epoch [12/30], Step [800/938], Loss: 0.0011\n",
            "Epoch [13/30], Step [400/938], Loss: 0.0221\n",
            "Epoch [13/30], Step [800/938], Loss: 0.0421\n",
            "Epoch [14/30], Step [400/938], Loss: 0.0032\n",
            "Epoch [14/30], Step [800/938], Loss: 0.0059\n",
            "Epoch [15/30], Step [400/938], Loss: 0.0036\n",
            "Epoch [15/30], Step [800/938], Loss: 0.0000\n",
            "Epoch [16/30], Step [400/938], Loss: 0.0000\n",
            "Epoch [16/30], Step [800/938], Loss: 0.0059\n",
            "Epoch [17/30], Step [400/938], Loss: 0.0016\n",
            "Epoch [17/30], Step [800/938], Loss: 0.0001\n",
            "Epoch [18/30], Step [400/938], Loss: 0.0006\n",
            "Epoch [18/30], Step [800/938], Loss: 0.0001\n",
            "Epoch [19/30], Step [400/938], Loss: 0.0001\n",
            "Epoch [19/30], Step [800/938], Loss: 0.0032\n",
            "Epoch [20/30], Step [400/938], Loss: 0.0004\n",
            "Epoch [20/30], Step [800/938], Loss: 0.0001\n",
            "Epoch [21/30], Step [400/938], Loss: 0.0021\n",
            "Epoch [21/30], Step [800/938], Loss: 0.0004\n",
            "Epoch [22/30], Step [400/938], Loss: 0.0050\n",
            "Epoch [22/30], Step [800/938], Loss: 0.0082\n",
            "Epoch [23/30], Step [400/938], Loss: 0.0019\n",
            "Epoch [23/30], Step [800/938], Loss: 0.0000\n",
            "Epoch [24/30], Step [400/938], Loss: 0.0072\n",
            "Epoch [24/30], Step [800/938], Loss: 0.0000\n",
            "Epoch [25/30], Step [400/938], Loss: 0.0002\n",
            "Epoch [25/30], Step [800/938], Loss: 0.0003\n",
            "Epoch [26/30], Step [400/938], Loss: 0.0723\n",
            "Epoch [26/30], Step [800/938], Loss: 0.0000\n",
            "Epoch [27/30], Step [400/938], Loss: 0.0012\n",
            "Epoch [27/30], Step [800/938], Loss: 0.0442\n",
            "Epoch [28/30], Step [400/938], Loss: 0.0257\n",
            "Epoch [28/30], Step [800/938], Loss: 0.0037\n",
            "Epoch [29/30], Step [400/938], Loss: 0.0000\n",
            "Epoch [29/30], Step [800/938], Loss: 0.0000\n",
            "Epoch [30/30], Step [400/938], Loss: 0.0001\n",
            "Epoch [30/30], Step [800/938], Loss: 0.0021\n"
          ]
        }
      ],
      "source": [
        "total_training_loss = train(model, criterion, optimizer, num_epochs=30)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KrAEczZ6DAWx",
        "outputId": "9e683e9f-e5e3-4d80-99cb-ad55338f0657"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[10.568266467943882,\n",
              " 3.527540081909407,\n",
              " 2.6088559687145545,\n",
              " 2.1823721174309565,\n",
              " 1.924547830021688,\n",
              " 1.6057489041481285,\n",
              " 1.3537017107074425,\n",
              " 1.1566425667095708,\n",
              " 1.1153176984784983,\n",
              " 0.983036599652807,\n",
              " 0.8899051300895366,\n",
              " 0.6929740591073604,\n",
              " 0.7318501428482301,\n",
              " 0.7322920497585915,\n",
              " 0.4553440813554328,\n",
              " 0.6452689993842532,\n",
              " 0.5951177723426396,\n",
              " 0.46549444908607823,\n",
              " 0.48881708133601715,\n",
              " 0.4589629755210131,\n",
              " 0.40898956688392407,\n",
              " 0.433147037635138,\n",
              " 0.3640874308061105,\n",
              " 0.3657881085964735,\n",
              " 0.3087785066473603,\n",
              " 0.3707788242852709,\n",
              " 0.24622227986580741,\n",
              " 0.282929215015277,\n",
              " 0.3211896866044501,\n",
              " 0.24679545621306762]"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "total_training_loss"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 278
        },
        "id": "gVWgnhLgDACb",
        "outputId": "48899b3d-1325-437f-ea99-d7b4e7a2e617"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3RU9bn/8fcDieES5RYQBSqgFASk2AZE9GjwwlHkFD2K1aL10h6tB0Xpr4jVVYtdUi+tWmm11lZae1TUini0tmJFIVY5argpN6vlIlGEJEIgChjC8/vjOyHhnpDM7Mzsz2utvWZm75nsZzP67O98r+buiIhIfDSLOgAREUktJX4RkZhR4hcRiRklfhGRmFHiFxGJmayoA6iLvLw87969e9RhiIiklXnz5pW6e8fd96dF4u/evTtFRUVRhyEiklbMbPXe9quqR0QkZpT4RURiRolfRCRm0qKOX0SalsrKSoqLi9m6dWvUoQjQokULunbtSnZ2dp3er8QvIvVWXFzMoYceSvfu3TGzqMOJNXenrKyM4uJievToUafPqKpHROpt69atdOjQQUm/CTAzOnToUK9fX0r8InJQlPSbjvp+F0r8IiIxk9mJ/7bb4NvfjjoKEWlkZWVlDBw4kIEDB9K5c2e6dOmy8/WXX365388WFRUxbty4A55j6NChjRLr7NmzGTlyZKP8rcaS2Y27H30EhYVRRyEijaxDhw4sXLgQgEmTJpGbm8sPf/jDnce3b99OVtbe01t+fj75+fkHPMebb77ZOME2QZld4s/Lg9JS0CpjIhnv8ssv5/vf/z4nnHACN954I2+//TYnnngixx9/PEOHDuX9998Hdi2BT5o0iSuvvJKCggJ69uzJlClTdv693Nzcne8vKCjgggsuoE+fPowZM4bqlQv/+te/0qdPH77xjW8wbty4epXsp02bxnHHHUf//v2ZOHEiAFVVVVx++eX079+f4447jvvuuw+AKVOm0LdvXwYMGMBFF13U4H+rzC7x5+XBtm3w+eeQ+BJFJAkKCvbcd+GF8N//DV98ASNG7Hn88svDVloKF1yw67HZsw8qjOLiYt58802aN2/Opk2beP3118nKyuKVV17h5ptvZvr06Xt8Zvny5bz22mts3ryZ3r17c8011+zRH37BggUsWbKEI488kpNOOok33niD/Px8rr76agoLC+nRowcXX3xxneP85JNPmDhxIvPmzaNdu3YMHz6c5557jm7duvHxxx+zePFiADZu3AjAnXfeycqVK8nJydm5ryEyv8QP4T8sEcl4o0ePpnnz5gCUl5czevRo+vfvz/jx41myZMleP3POOeeQk5NDXl4enTp1Yt26dXu8Z/DgwXTt2pVmzZoxcOBAVq1axfLly+nZs+fOvvP1SfzvvPMOBQUFdOzYkaysLMaMGUNhYSE9e/ZkxYoVXHfddbz00kscdthhAAwYMIAxY8bw2GOP7bMKqz4yu8TfvTuceCJUVkYdiUhm218JvVWr/R/PyzvoEv7uWrduvfP5j3/8Y4YNG8aMGTNYtWoVBXv7VQLk5OTsfN68eXO2b99+UO9pDO3atWPRokXMnDmThx56iKeffpqpU6fy4osvUlhYyAsvvMDkyZN57733GnQDyOwS/7Bh8Oab0KtX1JGISIqVl5fTpUsXAP74xz82+t/v3bs3K1asYNWqVQA89dRTdf7s4MGDmTNnDqWlpVRVVTFt2jROPfVUSktL2bFjB+effz6333478+fPZ8eOHaxZs4Zhw4Zx1113UV5eTkVFRYNiz+wSv4jE1o033shll13G7bffzjnnnNPof79ly5Y8+OCDnHXWWbRu3ZpBgwbt872zZs2ia9euO1//+c9/5s4772TYsGG4O+eccw6jRo1i0aJFXHHFFezYsQOAO+64g6qqKi655BLKy8txd8aNG0fbtm0bFLt5knq8mNlUYCSw3t37J/a1B54CugOrgAvdfcOB/lZ+fr4f1EIsFRUwdChcfz1897v1/7yI7NWyZcs49thjow4jchUVFeTm5uLujB07ll69ejF+/PhIYtnbd2Jm89x9j76ryazq+SNw1m77bgJmuXsvYFbidfK0agVLl0Lip5iISGP63e9+x8CBA+nXrx/l5eVcffXVUYdUJ0mr6nH3QjPrvtvuUUBB4vmjwGxgYrJioFkz6NBBvXpEJCnGjx8fWQm/IVLduHu4u69NPP8UOHxfbzSzq8ysyMyKSkpKDv6MSvwiSZGsamKpv/p+F5H16vEQ6T6jdfeH3T3f3fM7dtxjkfi6y8uDhtw4RGQPLVq0oKysTMm/Caiej79FixZ1/kyqe/WsM7Mj3H2tmR0BrE/6GQsKQiOviDSarl27UlxcTIN+jUujqV6Bq65SnfifBy4D7kw8/m/Sz/jTnyb9FCJxk52dXefVnqTpSVpVj5lNA+YCvc2s2My+S0j4Z5rZB8AZidciIpJCyezVs6+JK05P1jn3aupUmDABVqyANm1SemoRkaYos6dsAMjKgs8+U88eEZGEzE/8mqFTRGQXSvwiIjGjxC8iEjOZn/g7dQoLrnfrFnUkIiJNQuZPy5ybC48/HnUUIiJNRuaX+KtVVUUdgYhIkxCPxH/CCTB6dNRRiIg0CfFI/K1aqXFXRCQhHok/L0+JX0QkQYlfRCRm4pP4y8ogsYCxiEicZX53ToBTToFt26CyEnJyoo5GRCRS8Uj8Z54ZNhERiUlVjzts3hxK/SIiMRePxD9vHhx2GMycGXUkIiKRi0fi10RtIiI7xSvxl5VFG4eISBMQj8TfujW0aKESv4gIcUn8ZhrEJSKSEI/unAATJ8JRR0UdhYhI5OKT+K+9NuoIRESahHhU9QCUl8OKFVFHISISufgk/ltugUGDoo5CRCRy8Un8eXmwYQNs3x51JCIikYpX4ncPyV9EJMbilfhBXTpFJPaU+EVEYiY+ib9fP3jwQejZM+pIREQiFUniN7PxZrbEzBab2TQza5H0kx5xBFxzDXTpkvRTiYg0ZSlP/GbWBRgH5Lt7f6A5cFHST+wOixbB6tVJP5WISFMWVVVPFtDSzLKAVsAnST+jGQwdClOmJP1UIiJNWcoTv7t/DPwC+AhYC5S7+8u7v8/MrjKzIjMrKikpaZyTa6I2EZFIqnraAaOAHsCRQGszu2T397n7w+6e7+75HTt2bJyT5+VpTn4Rib0oqnrOAFa6e4m7VwLPAkNTcuaOHVXiF5HYiyLxfwQMMbNWZmbA6cCylJxZVT0iIqmfltnd3zKzZ4D5wHZgAfBwSk5+7bUwZkxKTiUi0lRFMh+/u/8E+EnKTzxkSMpPKSLS1MRn5C7AunXwt7/B559HHYmISGTilfgLC2HECFi5MupIREQiE6/Er4naRERilvg7dAiPSvwiEmPxSvwq8YuIxCzxq8QvIhJNd87I5OTAzJnQp0/UkYiIRCZeiR9g+PCoIxARiVS8qnoA5swJfflFRGIqfiX+u+6C9evh7LOjjkREJBLxK/FramYRibl4Jn716hGRGItn4q+ogK1bo45ERCQS8Uz8oOoeEYmt+CX+886Dd98Nq3GJiMRQ/Hr1dOyopC8isRa/Ev+mTfDAA7B4cdSRiIhEIn6Jf+vWsATjnDlRRyIiEon4Jf727cOjunSKSEzFL/FnZUHbtkr8IhJb8Uv8oEFcIhJrSvwiIjETv+6cANOnQ8uWUUchIhKJeCb+I4+MOgIRkcjEs6qnsBB+/GNwjzoSEZGUi2finzsXbr8dvvgi6khERFIunolfE7WJSIzFO/GrZ4+IxJASv4hIzCjxi4jETCTdOc2sLfB7oD/gwJXuPjdlARxzDGzeDK1bp+yUIiJNRVT9+O8HXnL3C8zsEKBVSs/evDnk5qb0lCIiTUXKq3rMrA1wCvAIgLt/6e4bUx0Ht90Gf/pTyk8rIhK1KOr4ewAlwB/MbIGZ/d7M9qhzMbOrzKzIzIpKSkoaP4onn4S//KXx/66ISBMXReLPAr4O/Mbdjwc+B27a/U3u/rC757t7fsdkLJXYoYMad0UklqJI/MVAsbu/lXj9DOFGkFqaoVNEYqpOid/MWptZs8Tzr5rZN80s+2BO6O6fAmvMrHdi1+nA0oP5Ww2ixC8iMVXXEn8h0MLMugAvA5cCf2zAea8DHjezd4GBwM8a8LcOTl4eVFZqojYRiZ26Jn5z9y+A/wQedPfRQL+DPam7L0zU3w9w93PdfcPB/q2DdscdUFICZik/tYhIlOqc+M3sRGAM8GJiX/PkhJQiSvgiElN1Tfw3AD8CZrj7EjPrCbyWvLBSYMkSuPhiWL486khERFKqTonf3ee4+zfd/a5EI2+pu49LcmzJtXlz6Mu/cmXUkYiIpFRde/U8YWaHJQZaLQaWmtmE5IaWZJqoTURiqq5VPX3dfRNwLvA3wujbS5MWVSoo8YtITNU18Wcn+u2fCzzv7pWEWTXTV5s2YbI2JX4RiZm6Jv7fAquA1kChmR0FbEpWUClhBr17h+QvIhIjdZqW2d2nAFNq7VptZsOSE1IKLVkSdQQiIilX18bdNmZ2b/VsmWZ2D6H0LyIiaaauVT1Tgc3AhYltE/CHZAWVMj//OXznO1FHISKSUnVdgetodz+/1uvbzGxhMgJKqQ8/hJkzo45CRCSl6lri32JmJ1e/MLOTgC3JCSmF8vKgrAx27Ig6EhGRlKlrif/7wJ8SyyYCbAAuS05IKZSXB1VVUF4O7dpFHY2ISErUdcqGRe7+NWAAMCCxctZpSY0sFTSIS0RiqF4rcLn7psQIXoAfJCGe1OrWDb72tTAvv4hITNS1qmdv0n9e44ICWJj+bdQiIvXRkDV303vKBhGRmNpv4jezzWa2aS/bZuDIFMWYPFu3wpAh8MgjUUciIpIy+63qcfdDUxVIJHJyYMEC+OCDqCMREUmZhlT1pD+z0LNHvXpEJEbinfhBiV9EYkeJX4lfRGKmId05M8OQIUr8IhIrSvyTJ0cdgYhISqmqR0QkZpT4p00LUzeoukdEYkKJ3x2Ki5X4RSQ2lPg1Q6eIxIwSvxK/iMRMZInfzJqb2QIz+0tUMQBK/CISO1GW+K8HlkV4/iAvD849F7p0iToSEZGUiKQfv5l1Bc4BJhP1gi6tWsGMGZGGICKSSlGV+H8J3Ajsc5VzM7vKzIrMrKikpCR1kYmIZLiUJ34zGwmsd/d5+3ufuz/s7vnunt+xY8fkBnXGGaG6R0QkBqIo8Z8EfNPMVgFPAqeZ2WMRxFGjWTNYty7SEEREUiXlid/df+TuXd29O3AR8Kq7X5LqOHahGTpFJEbUjx+U+EUkViKdndPdZwOzo4wBCIl/40aorITs7KijERFJKpX4IczJP3ZsSPwiIhlO8/EDDB8eNhGRGFCJv1plJWzdGnUUIiJJp8QPsH49tG0Ll14adSQiIkmnxA/QqRPcdBM88wy88krU0YiIJJUSf7UJE6BnT7juOvjyy6ijERFJGiX+ai1awP33w/Ll4VFEJEMp8dc2cmTYlkU/W7SISLKoO+funnkGcnKijkJEJGlU4t9dddJftgzm7XcCURGRtKQS/97s2AGjRoWbwPz5msZBRDKKSvx706wZ3H03LF4MDz4YdTQiIo1KiX9fRo2Cs86CW2+FTz+NOhoRkUajxL8vZjBlSpjGYeLEqKMREWk0quPfn1694Ec/CsnfPdwMRETSnBL/gUyaFHUEIiKNSlU9dfX3v8OTT0YdhYhIgynx14U73HUXXHMNlJREHY2ISIMo8deFGfzqV1BREWbxFBFJY0r8dXXssXDDDTB1Krz1VtTRiIgcNCX++rj1VjjySBgzBrZsiToaEZGDol499XHoofDss6HE37Jl1NGIiBwUlfjr64QTYNy48PzVV+Haa2HbtmhjEhGpByX+hpg7Fx54AE49FdasiToaEZE6UeJviFtuCfP3L10KX/86zJoVdUQiIgekxN9Q558P77wTFmwfPlw9fkSkyVPjbmPo3Tsk/KlTYfDgsE9z+4hIE6USf2PJzQ2NvmawYgUMGQLvvht1VCIie1DiT4aystDYO2QI3HgjrF0bdUQiIjulPPGbWTcze83MlprZEjO7PtUxJN2gQWHJxnPPhXvuge7dw6hfEZEmIIoS/3bg/7l7X2AIMNbM+kYQR3J17gxPPAH//CdccUVYzrHa++9HF5eIxF7KE7+7r3X3+Ynnm4FlQJdUx5EyRx8NDz0E994bXr/+OvTpAyNGQGFhaAQWEUmhSOv4zaw7cDywRx9IM7vKzIrMrKgkk6ZCPu44mDwZiorCwK+TT4YXXoAdO6KOTERiwjyiEqeZ5QJzgMnu/uz+3pufn+9FRUWpCSxVtmwJ3T9//vMw3fPq1dC6ddRRiUgGMbN57p6/+/5ISvxmlg1MBx4/UNLPWC1bwtix8MEHMHt2SPrbt4eF3TX9g4gkURS9egx4BFjm7vem+vxNTnY29O8fni9cCFOmhAFhP/2ppn4WkaSIosR/EnApcJqZLUxsIyKIo+nJz4fly2HkSPjJT8LiL888owZgEWlUkdXx10dG1vEfyOzZcP31UFkJixaFXwYiIvXQpOr4pQ4KCmDePHjppZD0N20Ko4DLyqKOTETSnBJ/U5aVBV/5Snj+2mthLECvXnDbbVBaGm1sIpK2lPjTxahRofH35JNh0qRwQ7juOqiqijoyEUkzSvzppH9/eP55WLIELroIPvoImjcPx1avjjY2EUkbSvzpqG/fMPhrxozwevXqMDXEGWfAzJnqBSQi+6XEn86qJ35r1w7uuAOWLYOzzoLjj4fHHw89gkREdqPEnwkOOwwmTAgLwEydGhL+5ZfDp5+G4+Xl+hUgIjsp8WeSnJwwBfR774V1gLt1C/vPOw/69YOf/QxWrYo0RBGJnhJ/JmrWDAYOrHn97W9DXh7ccgv06AH/9m/w3HPRxScikVLij4PvfS/M/b9yZZgSurS0phfQ5s3w1FNhsNjq1WGmUFULiWQ0TdkQR+6h/39WFjz2GFx66a7HDzkEXn0VTjopPD74IHToELZjjoFTTgm9iMyiiV9E6mRfUzZkRRGMRMwsJH0I4wF69QoNwWVl4ddAWVnNiOENG2Dp0rCvrKxmwNjSpWESufffD43JffvuurykiDRZSvxxl5UFJ5yw7+Pnnx82CKuELV8Ob74Zlo+EsJDMI49A+/ZhVPEpp4Q2hEGD9ItApIlSVY80zOrVYSbR118P7QgffBCqgT78MBy/995QtdS/f+hZ1KWLbggiKaKqHkmOo46Cyy4LG8DatVBcXHP8scdgwYKa123awHe+ExacAfi//wtVRm3apC5mkZhT4pfGdcQRYas2f35oN1iyBBYvDo/HHBOOVVaGqqFmzeA//gMuuQTOPjs0LotI0qiqR6JTWRmqiV58EaZNg/XrQ1vB1KlhNlIRaRBV9UjTk50NZ54Ztl/8Al55JVQN9e4djr/2WuhOOmZMTWPygWzfXtNjac0a+PLLXccltGwZ2hkgjGLOyoKuXRvtkkTSgfrfSdOQlRUmmHvssZokP3dumGbi2GNDL6E77oBHH635zIQJYaWyAQNC8m7VCk47reb42WeHaqVevWq2K6+sOX7qqWFai3//99DWIBITKvFL03XzzWHuoSefDDeEm28OPYaqG5I3bgxdTI8+OtwY2rev+bUA4aZRXh6eV/ckqt3+8MtfhjaH+++HE08MN4q77w49kA7Wtm1hwNthh8Hhh0PnzuHx8MPVdiFNhur4JX2UloZSfatWjft3Kyrg178OYxL+8pdwE6iqqlnk5kA++CBMif3Nb4Zqpd69w77aLr4YnngiPB8xItwYOneGr341jHvo108D4KTRqY5f0l9eXnL+bm4u3HQTXH99aAMAGDs2jGaeNGnXCe+qrVkT5jiaNi30XGrfHtatC1VW8+fDZ5+Fz69bFx6POip8rrIyzI/04Yeh62tFRdg/YUL4tVFZCW+/Dfn5YbZVkSRQ4hepVp30Abp3D1VMxx8fprWeNCm0JQDcdx/84Afh+aBBcM898K1v1TQq5+aGrXrai9qys8NgNwi/DlatCq/79Qv75s8PI6BzcmDw4PBr4OSTw2Nubqi6Ki6Gzz8P2xdfhMcRI8LxuXPhjTfCL4ra26BB4W9u2xZ+WWRnJ+EfUNKFqnpE9mXjxtAOcN99sGlT6GE0bFgYkPbii2Geo+oxCY1l0yaYNQv+8Y9wQ5g/P1Q7zZoVGq4ffhiuvnrPz1XPnVT7plTbJ5+E9o3bbgs3sZYtww2hU6fQ/jBjRrhx/OMfYRbX2u0TeXk11V6bN4ebT0VFzc3HPTSUQ/h3MQvTfx91VONXy0m97KuqR4lf5EA2bIAHHghVPiNHpvbcn38eehydeGJIov/6FxQVQevWu27HHBNK9FVVsGVLuIGUl4fHTZtC76fqXxtz5oR9GzaEsRMlJSHhN2sG//Vf8Pvf7xpDq1Yh0ZvBBRfA9Om7Hu/SpWa09tlnw0sv1Rzr1CkM0vvzn8PrF14IcfToEUZru4e427cPx9euDV1y3Wu2li3DDagxVFbCxx+Hf5s+fTK+Ok2JX0QObPPmkHzXratpn9iyJbRBALz8Mnz00a43nTZtQpsEhBvJhx+GKqzqrX17uPPOcLxXr5p5nKqNHBluCBB+lVQvGVpt9Gh4+unwvKAgnLNLFzjyyPCYnx+q5HbsCLGvWbPrNnIknH46vPtuuHlX57xDDgmfnTw5/F335M4jtX59+DfdsCFcw9FHQ9u2yTsfatwVkbo49NCwffWrez8+fPj+P9+pU9iGDt378dmzw81g5cqahu3qhm8IA/m2bAkJuHqr7qJbVRVi+/jj8Ktn/fqwf/z4kPhLS/ccjNeqVWivOf30cJ5bbw3vyc0Niw+98Qa0aBHe+8ILoZps6NCarV+/mmqu7dtD9d+GDTXb5s3hVxDA//xPGHT42Wfh2GefhV838+eH49/7Xs0Nrtpxx4UbUvXnv/wy3BCOPjrc1JLU00slfhFJT5WVoYSfnR1+KbjDb38bBuV17Roe27Wreym+sDC06bzxRs1N5dBDQym9ZUu44YYw5qM2s3BDaNYsHJ8+PfzCadcuPHbuHMZ1QLjplZaGY5s2hWq7Zs1q2mQGDAjrZVfLyYFf/SpUvx0kVfWIiNSFe/hF8sYboZfUHXeE6qzZs0PpvF27msTerl34RdIYJfPt20M12r/+BStWhMfzzgvtOwepSSV+MzsLuB9oDvze3e/c3/uV+EVE6m9fiT/lQwXNrDnwAHA20Be42Mz6pjoOEZG4imKM+GDgQ3df4e5fAk8CmoNXRCRFokj8XYA1tV4XJ/aJiEgKNNlZoczsKjMrMrOikpKSqMMREckYUST+j4FutV53Tezbhbs/7O757p7fsWPHlAUnIpLpokj87wC9zKyHmR0CXAQ8H0EcIiKxlPKRu+6+3cyuBWYSunNOdfclqY5DRCSuIpmywd3/Cvw1inOLiMRdWozcNbMSYPVuu/OA0gjCSZZMux7IvGvS9TR9mXZNDb2eo9x9j0bStEj8e2NmRXsbkZauMu16IPOuSdfT9GXaNSXreppsd04REUkOJX4RkZhJ58T/cNQBNLJMux7IvGvS9TR9mXZNSbmetK3jFxGRg5POJX4RETkISvwiIjGTdonfzM4ys/fN7EMzuynqeBqDma0ys/fMbKGZpd2KM2Y21czWm9niWvvam9nfzeyDxGO7KGOsr31c0yQz+zjxPS00sxFRxlgfZtbNzF4zs6VmtsTMrk/sT8vvaT/Xk87fUQsze9vMFiWu6bbE/h5m9lYi5z2VmOqmYedKpzr+xCIu/wTOJEzn/A5wsbsvjTSwBjKzVUC+u6flwBMzOwWoAP7k7v0T++4GPnP3OxM36HbuPjHKOOtjH9c0Cahw919EGdvBMLMjgCPcfb6ZHQrMA84FLicNv6f9XM+FpO93ZEBrd68ws2zgH8D1wA+AZ939STN7CFjk7r9pyLnSrcSvRVyaIHcvBD7bbfco4NHE80cJ/1OmjX1cU9py97XuPj/xfDOwjLAORlp+T/u5nrTlQUXiZXZic+A04JnE/kb5jtIt8WfqIi4OvGxm88zsqqiDaSSHu/vaxPNPgcOjDKYRXWtm7yaqgtKiWmR3ZtYdOB54iwz4nna7Hkjj78jMmpvZQmA98HfgX8BGd9+eeEuj5Lx0S/yZ6mR3/zphHeKxiWqGjOGhPjF96hT37TfA0cBAYC1wT7Th1J+Z5QLTgRvcfVPtY+n4Pe3letL6O3L3KncfSFinZDDQJxnnSbfEX6dFXNKNu3+ceFwPzCB84eluXaIetro+dn3E8TSYu69L/I+5A/gdafY9JeqNpwOPu/uzid1p+z3t7XrS/Tuq5u4bgdeAE4G2ZlY9k3Kj5Lx0S/wZt4iLmbVONE5hZq2B4cDi/X8qLTwPXJZ4fhnwvxHG0iiqE2TCeaTR95RoOHwEWObu99Y6lJbf076uJ82/o45m1jbxvCWhE8sywg3ggsTbGuU7SqtePQCJ7lm/pGYRl8kRh9QgZtaTUMqHsD7CE+l2TWY2DSggTCG7DvgJ8BzwNPAVwpTaF7p72jSW7uOaCghVCA6sAq6uVT/epJnZycDrwHvAjsTumwn14mn3Pe3nei4mfb+jAYTG2+aEQvnT7v7TRI54EmgPLAAucfdtDTpXuiV+ERFpmHSr6hERkQZS4hcRiRklfhGRmFHiFxGJGSV+EZGYUeIXAcysqtaMjgsbc+ZXM+tee5ZPkahlHfgtIrGwJTFUXiTjqcQvsh+JtRLuTqyX8LaZHZPY393MXk1MBjbLzL6S2H+4mc1IzKm+yMyGJv5UczP7XWKe9ZcTIzNFIqHELxK03K2q51u1jpW7+3HArwmjxgF+BTzq7gOAx4Epif1TgDnu/jXg68CSxP5ewAPu3g/YCJyf5OsR2SeN3BUBzKzC3XP3sn8VcJq7r0hMCvapu3cws1LCQiCVif1r3T3PzEqArrWH1CemDf67u/dKvJ4IZLv77cm/MpE9qcQvcmC+j+f1UXtulSrUviYRUuIXObBv1Xqcm3j+JmF2WIAxhAnDAGYB18DORTXapCpIkbpSqUMkaJlY+ajaS+5e3aWznZm9Syi1X5zYdx3wBzObAJQAVzfiJpQAAABQSURBVCT2Xw88bGbfJZTsryEsCCLSZKiOX2Q/EnX8+e5eGnUsIo1FVT0iIjGjEr+ISMyoxC8iEjNK/CIiMaPELyISM0r8IiIxo8QvIhIz/x9On3wOiMO+jAAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          },
          "output_type": "display_data"
        }
      ],
      "source": [
        "\n",
        "epoch_count = range(1, len(total_training_loss) + 1)\n",
        "\n",
        "# Visualize loss history\n",
        "plt.plot(epoch_count, total_training_loss, 'r--')\n",
        "plt.legend(['Training Loss', 'Test Loss'])\n",
        "plt.xlabel('Epoch')\n",
        "plt.ylabel('Loss')\n",
        "plt.show();"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "machine_shape": "hm",
      "name": "LeNet_From_Scratch.ipynb",
      "provenance": []
    },
    "interpreter": {
      "hash": "36cf16204b8548560b1c020c4e8fb5b57f0e4c58016f52f2d4be01e192833930"
    },
    "kernelspec": {
      "display_name": "Python 3.9.10 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.10"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}