RestNet_18_on_MNIST_Ange_VALLI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-K_R21Nqtfb0"
   },
   "source": [
    "<a href=\"https://www.kaggle.com/code/angevalli/resnet18-on-mnist/script\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a> <a href=\"https://colab.research.google.com/drive/1ltT_Ihk5jQzlRLi3sJ7Qt706d8QXt4Vs\" target=\"_blank\"><img align=\"left\" alt=\"Colab\" title=\"Open In Colab\" src=\"https://colab.research.google.com/assets/colab-badge.svg\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MOIkLWmzKbMD"
   },
   "source": [
    "# Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uw-kIsCPKahH"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch.utils.tensorboard import SummaryWriter\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torch.autograd import Variable\n",
    "from torchvision import datasets, transforms, models\n",
    "import csv\n",
    "\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E5TD2Uof5qej"
   },
   "source": [
    "# Part 1 : First script using Numpy\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eItK-RI-wpCw"
   },
   "source": [
    "This first script describes a Convolutional Neural Network taking randomly generated data $x$ as inputs and updating weights thanks to module *torch.randn()*. We also generated random outputs, the labels $y$. This network has three layers : the input layer, the hidden layer and the output layer. It has two weights, $ω_1$ between input and hidden layer and $ω_2$ between hidden and output layer.\n",
    "\n",
    "Each iteration involves a forward pass and a backward pass. The aim of the forward pass is to computes the current prediction value $y_{pred}$ thanks to an activation function. Here, the chosen function is a [ReLU (Rectified Linear Unit)](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)).\n",
    "The value of $y_pred$ is used for computing the loss function $ℒ = ||y_{pred} - y||_2$.\n",
    "The last step is to compute the backward pass, which means to calculate the gradients of $w_1$ and $w_2$. This requires the computation of gradients of output prediction $y_{pred}$ and gradients of each activation functions and weights in the hidden layers. Here, we consider only one hidden layer so their is no intermediary weights to take into consideration (chain rule)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b76CqsXl6NSj",
    "outputId": "7509d5da-c10b-47c2-8aeb-fee84b0dd563"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "99 770.0751953125\n",
      "199 4.263453960418701\n",
      "299 0.03719189018011093\n",
      "399 0.0006322634872049093\n",
      "499 6.916717393323779e-05\n"
     ]
    }
   ],
   "source": [
    "# -*- coding: utf-8 -*-\n",
    "\n",
    "dtype = torch.float\n",
    "device = torch.device(\"cpu\")\n",
    "# device = torch.device(\"cuda:0\") # Uncomment this to run on GPU\n",
    "\n",
    "# N is batch size; D_in is input dimension;\n",
    "# H is hidden dimension; D_out is output dimension.\n",
    "N, D_in, H, D_out = 64, 1000, 100, 10\n",
    "\n",
    "# Create random input and output data\n",
    "x = torch.randn(N, D_in, device=device, dtype=dtype)\n",
    "y = torch.randn(N, D_out, device=device, dtype=dtype)\n",
    "\n",
    "# Randomly initialize weights\n",
    "w1 = torch.randn(D_in, H, device=device, dtype=dtype)\n",
    "w2 = torch.randn(H, D_out, device=device, dtype=dtype)\n",
    "\n",
    "learning_rate = 1e-6\n",
    "for t in range(500):\n",
    "    # Forward pass: compute predicted y\n",
    "    h = x.mm(w1)\n",
    "    h_relu = h.clamp(min=0)\n",
    "    y_pred = h_relu.mm(w2)\n",
    "\n",
    "    # Compute and print loss\n",
    "    loss = (y_pred - y).pow(2).sum().item()\n",
    "    if t % 100 == 99:\n",
    "        print(t, loss)\n",
    "\n",
    "    # Backprop to compute gradients of w1 and w2 with respect to loss\n",
    "    grad_y_pred = 2.0 * (y_pred - y)\n",
    "    grad_w2 = h_relu.t().mm(grad_y_pred)\n",
    "    grad_h_relu = grad_y_pred.mm(w2.t())\n",
    "    grad_h = grad_h_relu.clone()\n",
    "    grad_h[h < 0] = 0\n",
    "    grad_w1 = x.t().mm(grad_h)\n",
    "\n",
    "    # Update weights using gradient descent\n",
    "    w1 -= learning_rate * grad_w1\n",
    "    w2 -= learning_rate * grad_w2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "M4OkVCrImV4D"
   },
   "source": [
    "# Part 2 : script using nn.Module\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SCgpccFkQR2P"
   },
   "source": [
    "We compute the same network from precedent part using Pytorch module. Instead of generating random data for inputs $x$ and labels $y$, we import [MNIST dataset](http://yann.lecun.com/exdb/mnist/) from torchvision, which is the module from Pytorch to import datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zwcZ75diTgsX"
   },
   "outputs": [],
   "source": [
    "# We import MNIST dataset from torchvision, downloaded in subfolder MNIST torchvision\n",
    "mnist = datasets.MNIST(root='MNIST torchvision/', download = True)\n",
    "x = torch.FloatTensor(mnist.data.reshape(mnist.data.shape[0], 28 * 28).tolist())\n",
    "y = torch.LongTensor(mnist.targets.tolist())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "G6ZJFJ9KHEk-"
   },
   "outputs": [],
   "source": [
    "# We define the neural network in Pytorch, which requires to define a class with nn.Module inheritance.\n",
    "class Net(nn.Module):\n",
    "  def __init__(self, D_in, H, D_out):\n",
    "    super(Net, self).__init__()\n",
    "    self.FC1 = nn.Linear(D_in, H)\n",
    "    self.relu = nn.ReLU()\n",
    "    self.FC2 = nn.Linear(H, D_out)\n",
    "  def forward(self, x):\n",
    "    A0 = x\n",
    "    A1 = self.FC1(A0)\n",
    "    A2 = self.relu(A1)\n",
    "    A3 = self.FC2(A1)\n",
    "    A4 = torch.sigmoid(A3)\n",
    "    return A4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "trjHGU3OSuB7"
   },
   "source": [
    "We consider an other dataset than the one generated in part 1 so there are specifications to consider. The dimensions of the layers are different : the MNIST dataset is composed of images of size $28*28$ pixels. Therefore, the input dimension of hidden layer is $D_{in} = 28*28=784$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uKQlCKvzmtiY"
   },
   "outputs": [],
   "source": [
    "# We compute the model\n",
    "\"\"\"\n",
    "N : input dimension of input layer\n",
    "D_in : output dimension of input layer, which is also the input dimension of hidden layer\n",
    "H : output dimension of hidden layer, which is also the input dimension of output layer\n",
    "D_out : output dimension of output layer\n",
    "\"\"\"\n",
    "N, D_in, H, D_out = 10000, 784, 200, 10\n",
    "model = Net(D_in, H, D_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CXq61aYTmzzH"
   },
   "outputs": [],
   "source": [
    "criterion = nn.CrossEntropyLoss() # The loss function is cross entropy loss\n",
    "optimizer = optim.SGD(model.parameters(),lr=0.00005, weight_decay= 1e-3, momentum = 0.9) # The optimizer is a stochastic gradient descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "iLRy60lOm4z-",
    "outputId": "f2f7e6cc-b412-4d07-eb07-32fc140ddd2a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 1, loss 2.409241, accuracy 0.07791666666666666\n",
      "epoch 2, loss 2.390407, accuracy 0.07966666666666666\n",
      "epoch 3, loss 2.366583, accuracy 0.08221666666666666\n",
      "epoch 4, loss 2.340013, accuracy 0.085525\n",
      "epoch 5, loss 2.313879, accuracy 0.08947333333333334\n",
      "epoch 6, loss 2.290153, accuracy 0.09378611111111111\n",
      "epoch 7, loss 2.268213, accuracy 0.09829523809523809\n",
      "epoch 8, loss 2.247641, accuracy 0.10285\n",
      "epoch 9, loss 2.227880, accuracy 0.10736851851851852\n",
      "epoch 10, loss 2.207004, accuracy 0.11181833333333334\n",
      "epoch 11, loss 2.187290, accuracy 0.11612121212121213\n",
      "epoch 12, loss 2.168538, accuracy 0.12035972222222223\n",
      "epoch 13, loss 2.150850, accuracy 0.1244948717948718\n",
      "epoch 14, loss 2.134138, accuracy 0.12853333333333333\n",
      "epoch 15, loss 2.117911, accuracy 0.13251222222222223\n",
      "epoch 16, loss 2.102536, accuracy 0.13640208333333334\n",
      "epoch 17, loss 2.087545, accuracy 0.14016176470588235\n",
      "epoch 18, loss 2.073259, accuracy 0.14388425925925927\n",
      "epoch 19, loss 2.058774, accuracy 0.14754649122807018\n",
      "epoch 20, loss 2.043656, accuracy 0.15118916666666668\n"
     ]
    }
   ],
   "source": [
    "# Write the training\n",
    "# We save summaries in subfolder runs/\n",
    "nb_epochs = 20\n",
    "writer = SummaryWriter('runs')\n",
    "total = 0\n",
    "correct = 0\n",
    "\n",
    "for num_epoch in range(1, nb_epochs+1) :\n",
    "  for i in range(0, x.shape[0], N):\n",
    "    inputs = Variable(x[i:i+N])\n",
    "    labels = Variable(y[i:i+N])\n",
    "    # zeroes the gradient buffers of all parameters\n",
    "    optimizer.zero_grad() # re-init the gradients (otherwise they are cumulated)\n",
    "    outputs = model(inputs) # Forward pass: Compute predicted y by passing  x to the model\n",
    "\n",
    "    loss = criterion(outputs, labels) # Compute loss\n",
    "    loss.backward() # perform back-propagation\n",
    "    # Perform the training parameters update\n",
    "    optimizer.step() # update the weights\n",
    "\n",
    "    # accuracy\n",
    "    _, predicted = torch.max(outputs.data, 1)\n",
    "    total += labels.size(0)\n",
    "    correct += (predicted == labels).sum().item()\n",
    "    accuracy = correct / total\n",
    "    writer.add_scalar('Loss/train', loss, num_epoch)\n",
    "    writer.add_scalar('Accuracy/train', accuracy, num_epoch)\n",
    "\n",
    "  print('epoch {}, loss {:.6f}, accuracy {}'.format(num_epoch, loss.item(), accuracy))\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "j1Y_xuIGT55i"
   },
   "source": [
    "We obtain better results with a higher batch size, which is the input dimension of input layer. We consider here a value of $10$ $000$. It induces a value of $200$ for the output dimension of hidden layer. The learning rate has a value of $5*10^{-5}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FooPp1D6gAN9"
   },
   "source": [
    "For a better understanding of the trainings, we save the values of accuracy and loss function through the epochs and visualise the performances thanks to Tensorboard summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tCg4iuYcJpk_"
   },
   "outputs": [],
   "source": [
    "# Show summaries\n",
    "%load_ext tensorboard\n",
    "%tensorboard --logdir=runs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-rcoUlHr_ols"
   },
   "source": [
    "# Part 3 : ResNet 18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VAs5my9goDw4"
   },
   "source": [
    "This code has been inspired from https://www.kaggle.com/c/digit-recognizer to implement ResNet 18 algorithm to work on MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Zuo_YEAIcVdl"
   },
   "outputs": [],
   "source": [
    "# We import training data from the following github repository : https://github.com/wehrley/Kaggle-Digit-Recognizer/blob/master/train.csv\n",
    "!npx degit --force wehrley/Kaggle-Digit-Recognizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3kNlAA_SZIeb"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# We compute training dataset from MNIST dataset we downloaded in .csv\n",
    "train = pd.read_csv(\"train.csv\", dtype = np.float32)\n",
    "\n",
    "# We compute training dataset and test dataset by splitting\n",
    "# We define labels and attributes\n",
    "targets_numpy = train.label.values\n",
    "features_numpy = train.loc[:,train.columns != \"label\"].values/255\n",
    "\n",
    "# We split with 80% for training dataset et 20% for testing dataset. \n",
    "features_train, features_test, targets_train, targets_test = train_test_split(features_numpy,\n",
    "                                                                             targets_numpy,\n",
    "                                                                             test_size = 0.2,\n",
    "                                                                             random_state = 42) \n",
    "\n",
    "# We use Pytorch Variables for accumulating gradients. We must use tensors.\n",
    "featuresTrain = torch.from_numpy(features_train)\n",
    "targetsTrain = torch.from_numpy(targets_train).type(torch.LongTensor)\n",
    "featuresTest = torch.from_numpy(features_test)\n",
    "targetsTest = torch.from_numpy(targets_test).type(torch.LongTensor)\n",
    "\n",
    "# Pytorch train and test sets\n",
    "train = torch.utils.data.TensorDataset(featuresTrain,targetsTrain)\n",
    "test = torch.utils.data.TensorDataset(featuresTest,targetsTest)\n",
    "\n",
    "# We use batch of size 100\n",
    "batch_size = 100\n",
    "\n",
    "# We load data using DataLoader of Pytorch\n",
    "train_loader = torch.utils.data.DataLoader(train, batch_size = batch_size, shuffle = False)\n",
    "test_loader = torch.utils.data.DataLoader(test, batch_size = batch_size, shuffle = False)\n",
    "\n",
    "# For first training, we define 2500 iterations, 7 epochs for our data and 0.05 learning_rate.\n",
    "n_iters = 2500\n",
    "num_epochs = n_iters / (len(features_train) / batch_size)\n",
    "num_epochs = int(num_epochs)\n",
    "learning_rate = 0.05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BUpfk73pZUzN"
   },
   "outputs": [],
   "source": [
    "# We define 3*3 convolutional neural network 3*3\n",
    "def conv3x3(in_channels, out_channels, stride=1):\n",
    "    return nn.Conv2d(in_channels, out_channels, kernel_size=3,\n",
    "                    stride=stride, padding=1, bias=False)\n",
    "\n",
    "\n",
    "# We define the residual block class useful for defining ResNet 18\n",
    "class ResidualBlock(nn.Module):\n",
    "    def __init__(self, in_channels, out_channels, stride=1, downsample=None):\n",
    "        super(ResidualBlock, self).__init__()\n",
    "        self.conv1 = conv3x3(in_channels, out_channels, stride)\n",
    "        self.bn1 = nn.BatchNorm2d(out_channels)\n",
    "        self.relu = nn.ReLU(inplace=True)\n",
    "        self.conv2 = conv3x3(out_channels, out_channels)\n",
    "        self.bn2 = nn.BatchNorm2d(out_channels)\n",
    "        self.downsample = downsample\n",
    "\n",
    "    def forward(self, x):\n",
    "        residual = x\n",
    "        out = self.conv1(x)\n",
    "        out = self.bn1(out)\n",
    "        out = self.relu(out)\n",
    "        out = self.conv2(out)\n",
    "        out = self.bn2(out)\n",
    "        if self.downsample:\n",
    "            residual = self.downsample(x)\n",
    "        out += residual\n",
    "        out = self.relu(out)\n",
    "        return out\n",
    "\n",
    "\n",
    "# We define the class to instantiate models based on ResNet 18\n",
    "class ResNet(nn.Module):\n",
    "    def __init__(self, block, layers, num_classes=10):\n",
    "        super(ResNet, self).__init__()\n",
    "        self.in_channels = 16\n",
    "        self.conv = conv3x3(1, 16)\n",
    "        self.bn = nn.BatchNorm2d(16)\n",
    "        self.relu = nn.ReLU(inplace=True)\n",
    "        self.layer1 = self.make_layer(block, 16, layers[0])\n",
    "        self.layer2 = self.make_layer(block, 32, layers[0], 2)\n",
    "        self.layer3 = self.make_layer(block, 64, layers[1], 2)\n",
    "        self.avg_pool = nn.AvgPool2d(8)\n",
    "        self.fc = nn.Linear(64, num_classes)\n",
    "\n",
    "    def make_layer(self, block, out_channels, blocks, stride=1):\n",
    "        downsample = None\n",
    "        if (stride != 1) or (self.in_channels != out_channels):\n",
    "            downsample = nn.Sequential(\n",
    "                conv3x3(self.in_channels, out_channels, stride=stride),\n",
    "                nn.BatchNorm2d(out_channels))\n",
    "        layers = []\n",
    "        layers.append(block(self.in_channels, out_channels, stride, downsample))\n",
    "        self.in_channels = out_channels\n",
    "        for i in range(1, blocks):\n",
    "            layers.append(block(out_channels, out_channels))\n",
    "        return nn.Sequential(*layers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        out = self.conv(x)\n",
    "        out = self.bn(out)\n",
    "        out = self.layer1(out)\n",
    "        out = self.layer2(out)\n",
    "        out = self.layer3(out)\n",
    "        out = self.avg_pool(out)\n",
    "        out = out.view(out.size(0), -1)\n",
    "        out = self.fc(out)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ToPUJ7s8ZZ6P"
   },
   "outputs": [],
   "source": [
    "# We instatiate a model based on ResNet class we implemented\n",
    "net_args = {\n",
    "    \"block\": ResidualBlock,\n",
    "    \"layers\": [2, 2, 2, 2]\n",
    "}\n",
    "model = ResNet(**net_args)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sYBRA3VhZhud"
   },
   "outputs": [],
   "source": [
    "# We define criterion and optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.Adam(model.parameters(),lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9G5SogeSZ1H3",
    "outputId": "4609bcc3-afed-4f8b-e235-4442d00c5835"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration: 500  Loss: 2.302534818649292  Accuracy: 8.357142448425293 %\n",
      "Iteration: 1000  Loss: 2.309339761734009  Accuracy: 10.821428298950195 %\n",
      "Iteration: 1500  Loss: 2.320518732070923  Accuracy: 10.630952835083008 %\n",
      "Iteration: 2000  Loss: 2.2974584102630615  Accuracy: 10.821428298950195 %\n"
     ]
    }
   ],
   "source": [
    "# Training of the model. Variables loss_list, iteration_list and accuracy_list are useful to save values of prediction and loss on ranges of iteration\n",
    "loss_list = []\n",
    "iteration_list = []\n",
    "accuracy_list = []\n",
    "count = 0\n",
    "for epoch in range(num_epochs):\n",
    "    for i, (images, labels) in enumerate(train_loader):\n",
    "        train  = Variable(images.resize_(batch_size, 1, 32, 32))\n",
    "        labels = Variable(labels)           \n",
    "        # Clear gradients\n",
    "        optimizer.zero_grad()    \n",
    "        # Forward propagation\n",
    "        outputs = model(train)        \n",
    "        # Calculate softmax and ross entropy loss\n",
    "        loss = criterion(outputs, labels)        \n",
    "        # Calculating gradients\n",
    "        loss.backward()        \n",
    "        # Update parameters\n",
    "        optimizer.step()        \n",
    "        count += 1     \n",
    "        if count % 250 == 0:\n",
    "            # Calculate Accuracy         \n",
    "            correct = 0\n",
    "            total = 0\n",
    "            # Iterate through test dataset\n",
    "            for images, labels in test_loader:\n",
    "                images = Variable(images.resize_(batch_size, 1, 32, 32))                \n",
    "                # Forward propagation\n",
    "                outputs = model(images)                \n",
    "                # Get predictions from the maximum value\n",
    "                predicted = torch.max(outputs.data, 1)[1]                \n",
    "                # Total number of labels\n",
    "                total += labels.size(0)              \n",
    "                correct += (predicted == labels).sum()         \n",
    "            accuracy = 100 * correct / float(total)           \n",
    "            # store loss and iteration\n",
    "            loss_list.append(loss.data)\n",
    "            iteration_list.append(count)\n",
    "            accuracy_list.append(accuracy)\n",
    "            if count % 500 == 0:\n",
    "                # Print Loss\n",
    "                print('Iteration: {}  Loss: {}  Accuracy: {} %'.format(count, loss.item(), accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MZhT7Bm-h9Pr"
   },
   "source": [
    "We obtain better results than the ones of the previous part, but the computing time is higher. The network is more robust and show better results on the first epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yG4MgK1fi7uv"
   },
   "source": [
    "ResNet 18 computes regularisations to prevent the risk of overfitting on the data. This regularisation is computed by a normalisation layer which follows every convolutional layer in the network. It corresponds to the addition of some noise when updating the weights. This noise depends of the batch size but also of the mean and the standard deviation on batch values.\n",
    "Other methods exist to diminish the error du to generalisation in deep neural network, such as the [DropOut](https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "veFn5L6lphkg"
   },
   "source": [
    "## Grid Search for hyper parameters optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "g9bza9eX-zAz"
   },
   "outputs": [],
   "source": [
    "# We define a data_load to reload the data with the batch_size\n",
    "def data_load(batch_size) :\n",
    "  return torch.utils.data.DataLoader(train, batch_size = batch_size, shuffle = False), torch.utils.data.DataLoader(test, batch_size = batch_size, shuffle = False)\n",
    "\n",
    "# We use this function to create a new network, to train it using parameters and to recover precision after 250 iterations or end of DataLoader\n",
    "def resnet18_simplified_for_grid_search(optimizer, batch_size=64) :\n",
    "  net_args = {\n",
    "      \"block\": ResidualBlock,\n",
    "      \"layers\": [2, 2, 2, 2]\n",
    "  }\n",
    "  model = ResNet(**net_args)\n",
    "  train_loader, test_loader = data_load(batch_size)\n",
    "  count = 0\n",
    "  for i, (images, labels) in enumerate(train_loader):\n",
    "      train  = Variable(images.resize_(batch_size, 1, 32, 32))\n",
    "      labels = Variable(labels)           \n",
    "      # Clear gradients\n",
    "      optimizer.zero_grad()    \n",
    "      # Forward propagation\n",
    "      outputs = model(train)        \n",
    "      # Calculate softmax and ross entropy loss\n",
    "      loss = criterion(outputs, labels)        \n",
    "      # Calculating gradients\n",
    "      loss.backward()        \n",
    "      # Update parameters\n",
    "      optimizer.step()        \n",
    "      count += 1\n",
    "      if count % 250 == 0 or count == len(list(enumerate(train_loader))):\n",
    "            # Calculate Accuracy         \n",
    "            correct = 0\n",
    "            total = 0\n",
    "            # Iterate through test dataset\n",
    "            for images, labels in test_loader:\n",
    "                images = Variable(images.resize_(batch_size, 1, 32, 32))                \n",
    "                # Forward propagation\n",
    "                outputs = model(images)                \n",
    "                # Get predictions from the maximum value\n",
    "                predicted = torch.max(outputs.data, 1)[1]            \n",
    "                # Total number of labels\n",
    "                try:\n",
    "                  correct += (predicted == labels).sum() \n",
    "                  total += labels.size(0)\n",
    "                except:\n",
    "                  None               \n",
    "            accuracy = 100 * correct / float(total)\n",
    "            break\n",
    "  return accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HNfPErRH5m13"
   },
   "outputs": [],
   "source": [
    "# We define the values of parameters for performing grid search.\n",
    "learning_rate_list_1 = [1.0,0.1,0.01,0.001]\n",
    "weight_decay_list = [0.1,0.01,0.001]\n",
    "batch_size_1 = [5,200]\n",
    "\n",
    "learning_rate_list_2 = [0.11, 0.9910, 0.9905]\n",
    "batch_size_2 = [5,64,200]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "w6rHYiHR8z57"
   },
   "outputs": [],
   "source": [
    "# We write the result of grid search in grid_search.csv file\n",
    "f = open(\"grid_search.csv\", \"a\")\n",
    "for i in range(len(learning_rate_list_1)) :\n",
    "  for j in range(len(weight_decay_list)) :\n",
    "    optimizer = optim.Adam(model.parameters(),lr=learning_rate_list_1[i],weight_decay=weight_decay_list[j])\n",
    "    current_accuracy = resnet18_simplified_for_grid_search(optimizer)\n",
    "    f.write(\" Learning rate = {}, Weight Decay = {}, Batch_size = {}, Accuracy = {} \".format(learning_rate_list_1[i],weight_decay_list[j],64,current_accuracy) + '\\n')\n",
    "  for k in range(len(batch_size_1)) :\n",
    "    optimizer = optim.Adam(model.parameters(),lr=learning_rate_list_1[i],weight_decay=0.001)\n",
    "    current_accuracy = resnet18_simplified_for_grid_search(optimizer,batch_size=batch_size_1[k])\n",
    "    f.write(\" Learning rate = {}, Weight Decay = {}, Batch_size = {}, Accuracy = {} \".format(learning_rate_list_1[i],0.001,batch_size_1[k],current_accuracy) + '\\n')\n",
    "for i in range(len(learning_rate_list_2)) :\n",
    "  for k in range(len(batch_size_2)) :\n",
    "    optimizer = optim.Adam(model.parameters(),lr=learning_rate_list_2[i],weight_decay=0.001)\n",
    "    current_accuracy = resnet18_simplified_for_grid_search(optimizer,batch_size=batch_size_2[k])\n",
    "    f.write(\" Learning rate = {}, Weight Decay = {}, Batch_size = {}, Accuracy = {} \".format(learning_rate_list_2[i],0.001,batch_size_2[k],current_accuracy) + '\\n')\n",
    "f.close()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "RestNet_18_on_MNIST_Ange_VALLI.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}