code/05-multi-layer-perceptrons.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📝 **Author:** Amirhossein Heydari - 📧 **Email:** <amirhosseinheydari78@gmail.com> - 📍 **Origin:** [mr-pylin/pytorch-workshop](https://github.com/mr-pylin/pytorch-workshop)\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- [Dependencies](#toc1_)    \n",
    "- [Multilayer Perceptron (MLP)](#toc2_)    \n",
    "  - [Forward Propagation Using Linear Algebra](#toc2_1_)    \n",
    "  - [Gradient Computation and Backpropagation](#toc2_2_)    \n",
    "  - [Multilayer Perceptron Using PyTorch](#toc2_3_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=1\n",
    "\tmaxLevel=6\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc1_'></a>[Dependencies](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn.functional as F\n",
    "from torch import nn\n",
    "from torchinfo import summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set a seed for deterministic results\n",
    "seed = 42\n",
    "torch.manual_seed(seed)\n",
    "torch.backends.cudnn.deterministic = True\n",
    "torch.backends.cudnn.benchmark = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check if cuda is available\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# log\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc2_'></a>[Multilayer Perceptron (MLP)](#toc0_)\n",
    "\n",
    "- A [**Multilayer Perceptron (MLP)**](https://en.wikipedia.org/wiki/Multilayer_perceptron) is a type of feedforward artificial neural network, also known as a **Fully-Connected Network** or **Dense Network**.\n",
    "- It consists of at least three layers of nodes: an **input layer**, one or more **hidden layers**, and an **output layer**.\n",
    "\n",
    "🧬 **Key Characteristics**:\n",
    "\n",
    "- **Fully Connected**: Every node (neuron) in one layer is connected to every node in the next layer.\n",
    "- **Non-Linear [Activations](./utils/activation-functions.ipynb)**: Each neuron applies a non-linear activation function, enabling the network to model complex patterns.\n",
    "- **[Feedforward](https://en.wikipedia.org/wiki/Feedforward_neural_network)**: Data flows in a single direction, from input to output, with no cycles or loops.\n",
    "\n",
    "🏛️ **Basic Architecture**:\n",
    "\n",
    "- **Input Layer**: Receives input features. The number of neurons equals the number of features in the dataset.\n",
    "- **Hidden Layers**: These layers contain neurons that compute weighted sums and apply activation functions.\n",
    "- **Output Layer**: Produces the final output, which could be a single value or a set of values for different tasks e.g. [**regression**](https://en.wikipedia.org/wiki/Regression_analysis), and [**classification**](https://en.wikipedia.org/wiki/Classification).\n",
    "\n",
    "<figure style=\"text-align: center;\">\n",
    "  <img src=\"../assets/images/original/mlp/multi-layer-perceptrons.svg\" alt=\"multi-layer-perceptrons.svg\" style=\"width: 100%;\">\n",
    "  <figcaption style=\"text-align: center;\">Multi-Layer-Perceptron (aka fully connected layers)</figcaption>\n",
    "</figure>\n",
    "\n",
    "<table style=\"margin: 0 auto; text-align:center;\">\n",
    "  <thead>\n",
    "    <tr>\n",
    "      <th colspan=\"2\">hidden<sub>1</sub> parameters</th>\n",
    "      <th colspan=\"2\">hidden<sub>2</sub> parameters</th>\n",
    "      <th colspan=\"2\">logits parameters</th>\n",
    "    </tr>\n",
    "  </thead>\n",
    "  <tbody>\n",
    "    <tr>\n",
    "      <td>Weights</td>\n",
    "      <td>Biases</td>\n",
    "      <td>Weights</td>\n",
    "      <td>Biases</td>\n",
    "      <td>Weights</td>\n",
    "      <td>Biases</td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "      <td>A × B</td>\n",
    "      <td>B</td>\n",
    "      <td>B × C</td>\n",
    "      <td>C</td>\n",
    "      <td>C × D</td>\n",
    "      <td>D</td>\n",
    "    </tr>\n",
    "  </tbody>\n",
    "  <tfoot>\n",
    "    <tr>\n",
    "      <td colspan=\"2\">(A + 1) × B</td>\n",
    "      <td colspan=\"2\">(B + 1) × C</td>\n",
    "      <td colspan=\"2\">(C + 1) × D</td>\n",
    "    </tr>\n",
    "  </tfoot>\n",
    "</table>\n",
    "\n",
    "📉 **Limitations of MLPs**:\n",
    "\n",
    "- **Fixed Input and Output Sizes**:\n",
    "  - MLPs require a fixed size for both input and output, making them less flexible for tasks involving variable-length sequences.\n",
    "- **Lack of Temporal Awareness**:\n",
    "  - MLPs do not inherently handle temporal data well.\n",
    "  - They treat each input independently, which means they can't capture the temporal dependencies in sequential data.\n",
    "- **Scalability Issues**:\n",
    "  - As the size of the input data grows, the number of parameters in an MLP increases significantly, leading to higher computational costs and potential **overfitting**.\n",
    "- **Stateless Nature**:\n",
    "  - MLPs learn a fixed function approximation and do not maintain any state between inputs, which limits their ability to model dynamic processes.\n",
    "\n",
    "⚔️ **MLPs vs. Other Architectures**:\n",
    "\n",
    "- MLPs vs. [CNNs (Convolutional Neural Networks)](./08-convolutional-neural-networks.ipynb): CNNs are better suited for image data because they can capture spatial hierarchies, while MLPs are more general-purpose.\n",
    "- MLPs vs. [RNNs (Recurrent Neural Networks)](./12-recurrent-neural-networks.ipynb): RNNs are used for sequential data (e.g., time series, language modeling) because they can handle temporal dependencies.\n",
    "\n",
    "🛠️ **Weight and Bias Initialization**:\n",
    "\n",
    "- **Weight**\n",
    "  - Weights are initialized using the Kaiming (He) initialization by default, which is suitable for layers using ReLU activation functions.\n",
    "  - the weights are initialized from a uniform distribution with a range based on the number of input and output units.\n",
    "      $$W \\sim \\mathcal{U}\\left(-{gain}\\times\\sqrt{\\frac{6}{n_{\\text{in}} + n_{\\text{out}}}}, {gain}\\times\\sqrt{\\frac{6}{n_{\\text{in}} + n_{\\text{out}}}}\\right)$$\n",
    "- **Bias**:\n",
    "  - Biases are initialized to zero by default.\n",
    "- More Details about Initialization: [hyperparameters.ipynb](./utils/hyperparameters.ipynb)\n",
    "\n",
    "🛝 **Playgrounds**:\n",
    "\n",
    "- [deeperplayground.org](https://deeperplayground.org/)\n",
    "- [alexlenail.me/NN-SVG](https://alexlenail.me/NN-SVG/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_1_'></a>[Forward Propagation Using Linear Algebra](#toc0_)\n",
    "\n",
    "- **Layer 1 (First Hidden Layer)**\n",
    "  - **Input**: $x \\in ℝ^d$, where $d$ is the number of input features.\n",
    "  - **Weights**: $W^{(1)} \\in ℝ^{h_1 \\times d}$, where $h_1$​ is the number of neurons in the first hidden layer.\n",
    "  - **Biases**: $b^{(1)} \\in ℝ^{h_1}$.\n",
    "  - The transformation for the first hidden layer is:\n",
    "      $$\\mathbf{z}^{(1)} = \\sigma(\\mathbf{W}^{(1)} \\mathbf{x} + \\mathbf{b}^{(1)})$$\n",
    "\n",
    "- **For each subsequent layer** $l$, where $l = \\{2, 3, \\ldots, L − 1\\}$\n",
    "  - **Input** from the previous layer: $z^{(l-1)} \\in ℝ^{h_{l-1}}$.\n",
    "  - **Weights**: $W^{(l)} \\in ℝ^{h_l \\times h_{l-1}}$, where $h_l$​ is the number of neurons in the $l$-th hidden layer.\n",
    "  - **Biases**: $b^{(1)} \\in ℝ^{h_l}$.\n",
    "  - The transformation for each hidden layer is:\n",
    "      $$\\mathbf{z}^{(l)} = \\sigma(\\mathbf{W}^{(l)} \\mathbf{a}^{(l-1)} + \\mathbf{b}^{(l)})$$\n",
    "\n",
    "- **Output Layer**\n",
    "  - **Weights**: $W^{(L)} \\in ℝ^{o \\times h_{L-1}}$, where $o$ is the number of output neurons.\n",
    "  - **Biases**: $b^{(L)} \\in ℝ^{o}$.\n",
    "  - The transformation for the output is:\n",
    "      $$\\mathbf{\\hat{y}} = \\sigma_L(\\mathbf{W}^{(L)} \\mathbf{a}^{(L-1)} + \\mathbf{b}^{(L)})$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLP(torch.nn.Module):\n",
    "    def __init__(self, input_size: int, hidden_size1: int, hidden_size2: int, output_size: int):\n",
    "        super().__init__()\n",
    "\n",
    "        # initialize weights and biases for the first hidden layer\n",
    "        self.W1 = nn.Parameter(torch.randn(hidden_size1, input_size))\n",
    "        self.b1 = nn.Parameter(torch.randn(hidden_size1))\n",
    "\n",
    "        # initialize weights and biases for the second hidden layer\n",
    "        self.W2 = nn.Parameter(torch.randn(hidden_size2, hidden_size1))\n",
    "        self.b2 = nn.Parameter(torch.randn(hidden_size2))\n",
    "\n",
    "        # initialize weights and biases for the output layer\n",
    "        self.W3 = nn.Parameter(torch.randn(output_size, hidden_size2))\n",
    "        self.b3 = nn.Parameter(torch.randn(output_size))\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        self.z1 = x @ self.W1.T + self.b1\n",
    "        self.a1 = F.relu(self.z1)\n",
    "\n",
    "        self.z2 = self.a1 @ self.W2.T + self.b2\n",
    "        self.a2 = F.relu(self.z2)\n",
    "\n",
    "        self.z3 = self.a2 @ self.W3.T + self.b3\n",
    "        return self.z3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example input\n",
    "batch_size = 3\n",
    "x = torch.randn(batch_size, 10)\n",
    "y = torch.randn(batch_size, 2)\n",
    "\n",
    "# initialize the MLP\n",
    "input_size = 10  # number of input features\n",
    "hidden_size1 = 5  # number of neurons in the first hidden layer\n",
    "hidden_size2 = 3  # number of neurons in the second hidden layer\n",
    "output_size = 2  # number of output neurons (e.g., for binary classification)\n",
    "\n",
    "model_1 = MLP(input_size, hidden_size1, hidden_size2, output_size)\n",
    "model_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary(model_1, input_size=(x.size()), device=\"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform forward propagation\n",
    "with torch.no_grad():\n",
    "    y_pred = model_1.forward(x)\n",
    "\n",
    "# log\n",
    "print(f\"y_pred:\\n{y_pred}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_2_'></a>[Gradient Computation and Backpropagation](#toc0_)\n",
    "\n",
    "- **Compute the Loss**:\n",
    "   $$\\mathcal{L}(\\mathbf{\\hat{y}}, \\mathbf{y})$$\n",
    "- **Backpropagation**\n",
    "  - Compute the gradient of the loss with respect to the output layer weights and biases:\n",
    "      $$\\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{W}^{(L)}} = \\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{z}^{(L)}} \\cdot \\frac{\\partial \\mathbf{z}^{(L)}}{\\partial \\mathbf{W}^{(L)}}$$\n",
    "  - Compute gradients for the weights and biases of each preceding layer:\n",
    "      $$\\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{W}^{(l)}} = \\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{z}^{(l)}} \\cdot \\frac{\\partial \\mathbf{z}^{(l)}}{\\partial \\mathbf{W}^{(l)}}$$\n",
    "- **Update the Parameters**\n",
    "  - using a gradient-based optimization algorithm like Gradient Descent or Adam:\n",
    "   $$\\mathbf{W}^{(l)} = \\mathbf{W}^{(l)} - \\eta \\frac{\\partial \\mathcal{L}}{\\partial \\mathbf{W}^{(l)}}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backward(self, x: torch.Tensor, y: torch.Tensor, learning_rate: float) -> None:\n",
    "    # compute the loss (Mean Squared Error - MSE)\n",
    "    # loss = (1/N) * sum((z3 - y)^2) over all batch samples\n",
    "    loss = torch.mean((self.z3 - y) ** 2)\n",
    "\n",
    "    # compute the gradient of the loss with respect to z3 (output layer pre-activation)\n",
    "    # this is the local gradient for the loss function with respect to z3\n",
    "    # d(loss)/d(z3) = 2 * (z3 - y) / N\n",
    "    loss_grad = 2 * (self.z3 - y) / y.size(0)\n",
    "\n",
    "    # compute the gradient of the loss with respect to W3 (weights between hidden layer 2 and output layer)\n",
    "    # d(loss)/d(W3) = d(loss)/d(z3) * d(z3)/d(W3)\n",
    "    # d(z3)/d(W3) = a2^T (activation of hidden layer 2)\n",
    "    # grad_W3 = (loss_grad)^T * a2\n",
    "    grad_W3 = torch.matmul(loss_grad.T, self.a2)\n",
    "\n",
    "    # compute the gradient of the loss with respect to b3 (biases of the output layer)\n",
    "    # d(loss)/d(b3) = d(loss)/d(z3) * d(z3)/d(b3)\n",
    "    # d(z3)/d(b3) = 1 (bias gradient accumulates over the batch dimension)\n",
    "    # grad_b3 = sum(loss_grad) across batch dimension\n",
    "    grad_b3 = torch.sum(loss_grad, dim=0)\n",
    "\n",
    "    # backpropagate the gradient to the second hidden layer (w.r.t. a2)\n",
    "    # compute the gradient of the loss with respect to a2 (activation of hidden layer 2)\n",
    "    # d(loss)/d(a2) = d(loss)/d(z3) * d(z3)/d(a2)\n",
    "    # d(z3)/d(a2) = W3 (weights between hidden layer 2 and output layer)\n",
    "    grad_a2 = torch.matmul(loss_grad, self.W3)\n",
    "\n",
    "    # compute the gradient of the loss with respect to z2 (pre-activation of hidden layer 2)\n",
    "    # this is the local gradient for ReLU at the second hidden layer\n",
    "    # d(z2)/d(a2) = ReLU'(z2) (element-wise derivative of ReLU)\n",
    "    # grad_z2 = grad_a2 * ReLU'(z2) (ReLU'(z2) is 1 where z2 > 0, else 0)\n",
    "    grad_z2 = grad_a2 * (self.a2 > 0).float()\n",
    "\n",
    "    # compute the gradient of the loss with respect to W2 (weights between hidden layer 1 and hidden layer 2)\n",
    "    # d(loss)/d(W2) = d(loss)/d(z2) * d(z2)/d(W2)\n",
    "    # d(z2)/d(W2) = a1^T (activation of hidden layer 1)\n",
    "    # grad_W2 = (grad_z2)^T * a1\n",
    "    grad_W2 = torch.matmul(grad_z2.T, self.a1)\n",
    "\n",
    "    # compute the gradient of the loss with respect to b2 (biases of hidden layer 2)\n",
    "    # d(loss)/d(b2) = d(loss)/d(z2) * d(z2)/d(b2)\n",
    "    # d(z2)/d(b2) = 1 (bias gradient accumulates over the batch dimension)\n",
    "    # grad_b2 = sum(grad_z2) across batch dimension\n",
    "    grad_b2 = torch.sum(grad_z2, dim=0)\n",
    "\n",
    "    # backpropagate the gradient to the first hidden layer (w.r.t. a1)\n",
    "    # compute the gradient of the loss with respect to a1 (activation of hidden layer 1)\n",
    "    # d(loss)/d(a1) = d(loss)/d(z2) * d(z2)/d(a1)\n",
    "    # d(z2)/d(a1) = W2 (weights between hidden layer 1 and hidden layer 2)\n",
    "    grad_a1 = torch.matmul(grad_z2, self.W2)\n",
    "\n",
    "    # compute the gradient of the loss with respect to z1 (pre-activation of hidden layer 1)\n",
    "    # this is the local gradient for ReLU at the first hidden layer\n",
    "    # d(z1)/d(a1) = ReLU'(z1) (element-wise derivative of ReLU)\n",
    "    # grad_z1 = grad_a1 * ReLU'(z1) (ReLU'(z1) is 1 where z1 > 0, else 0)\n",
    "    grad_z1 = grad_a1 * (self.a1 > 0).float()\n",
    "\n",
    "    # compute the gradient of the loss with respect to W1 (weights between input layer and hidden layer 1)\n",
    "    # d(loss)/d(W1) = d(loss)/d(z1) * d(z1)/d(W1)\n",
    "    # d(z1)/d(W1) = x^T (input features)\n",
    "    # grad_W1 = (grad_z1)^T * x\n",
    "    grad_W1 = torch.matmul(grad_z1.T, x)\n",
    "\n",
    "    # compute the gradient of the loss with respect to b1 (biases of hidden layer 1)\n",
    "    # d(loss)/d(b1) = d(loss)/d(z1) * d(z1)/d(b1)\n",
    "    # d(z1)/d(b1) = 1 (bias gradient accumulates over the batch dimension)\n",
    "    # grad_b1 = sum(grad_z1) across batch dimension\n",
    "    grad_b1 = torch.sum(grad_z1, dim=0)\n",
    "\n",
    "    # update parameters using gradients (Gradient Descent step)\n",
    "    with torch.no_grad():\n",
    "        self.W1 -= learning_rate * grad_W1\n",
    "        self.b1 -= learning_rate * grad_b1\n",
    "        self.W2 -= learning_rate * grad_W2\n",
    "        self.b2 -= learning_rate * grad_b2\n",
    "        self.W3 -= learning_rate * grad_W3\n",
    "        self.b3 -= learning_rate * grad_b3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "MLP.backward = backward"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example input\n",
    "batch_size = 3\n",
    "x = torch.randn(batch_size, 10)\n",
    "y = torch.randn(batch_size, 2)\n",
    "\n",
    "# initialize the MLP\n",
    "input_size = 10  # Number of input features\n",
    "hidden_size1 = 5  # Number of neurons in the first hidden layer\n",
    "hidden_size2 = 3  # Number of neurons in the second hidden layer\n",
    "output_size = 2  # Number of output neurons\n",
    "\n",
    "model_2 = MLP(input_size, hidden_size1, hidden_size2, output_size)\n",
    "summary(model_2, input_size=x.size(), device=\"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform forward propagation\n",
    "with torch.no_grad():\n",
    "    y_pred_1 = model_2.forward(x)\n",
    "\n",
    "# Perform backward propagation and update weights\n",
    "learning_rate = 0.01\n",
    "model_2.backward(x, y, learning_rate)\n",
    "\n",
    "# Perform forward propagation again to see updated output\n",
    "with torch.no_grad():\n",
    "    y_pred_2 = model_2.forward(x)\n",
    "\n",
    "# log\n",
    "print(f\"y_true:\\n{y}\\n\")\n",
    "print(f\"output before backpropagation:\\n{y_pred_1}\\n\")\n",
    "print(f\"output after backpropagation:\\n{y_pred_2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_3_'></a>[Multilayer Perceptron Using PyTorch](#toc0_)\n",
    "\n",
    "- Refer to this [**notebook**](./projects/01-multi-layer-perceptrons.ipynb) for a comprehensive example on the MLP concept.\n",
    "\n",
    "📚 **Tutorials**:\n",
    "\n",
    "- Neural Networks: [pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial](https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial)\n",
    "- Training a Classifier: [pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLP2(nn.Module):\n",
    "    def __init__(self, input_size: int, hidden_size1: int, hidden_size2: int, output_size: int):\n",
    "        super().__init__()\n",
    "        # define layers using nn.Linear\n",
    "        self.fc1 = nn.Linear(input_size, hidden_size1)  # first hidden layer\n",
    "        self.fc2 = nn.Linear(hidden_size1, hidden_size2)  # second hidden layer\n",
    "        self.fc3 = nn.Linear(hidden_size2, output_size)  # output layer\n",
    "\n",
    "        # define activation function (ReLU)\n",
    "        self.relu = nn.ReLU()\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        # forward pass through the network\n",
    "        x = self.relu(self.fc1(x))  # first hidden layer with ReLU\n",
    "        x = self.relu(self.fc2(x))  # second hidden layer with ReLU\n",
    "        x = self.fc3(x)  # output layer (no activation here)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_size = 500  # number of input features\n",
    "hidden_size1 = 10  # size of the first hidden layer\n",
    "hidden_size2 = 8  # size of the second hidden layer\n",
    "num_classes = 3  # number of output features\n",
    "\n",
    "# initialize the model\n",
    "model_3 = MLP2(input_size, hidden_size1, hidden_size2, num_classes)\n",
    "\n",
    "# log\n",
    "model_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# example input\n",
    "batch_size = 32\n",
    "x = torch.randn(batch_size, input_size)\n",
    "y = torch.randint(0, num_classes, (batch_size,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary(model_3, input_size=x.size(), device=\"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize criterion and optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(model_3.parameters(), lr=0.01)\n",
    "\n",
    "num_epochs = 100\n",
    "\n",
    "# training loop\n",
    "for epoch in range(num_epochs):\n",
    "\n",
    "    # forward pass\n",
    "    y_pred = model_3(x)\n",
    "\n",
    "    # compute the loss\n",
    "    loss = criterion(y_pred, y)\n",
    "\n",
    "    # perform backward propagation automatically\n",
    "    loss.backward()\n",
    "\n",
    "    # update the weights & zero the gradients\n",
    "    optimizer.step()\n",
    "    optimizer.zero_grad()\n",
    "\n",
    "    # compute accuracy\n",
    "    acc = (y_pred.argmax(dim=1) == y).sum().item() / batch_size\n",
    "\n",
    "    # log\n",
    "    if epoch % 10 == 0 or (epoch + 1) == num_epochs:\n",
    "        print(f\"epoch {epoch+1:0{len(str(num_epochs))}}/{num_epochs} -> loss: {loss.item():6.4f} | acc: {acc*100:5.2f}%\")"
   ]
  }
 ],
 "metadata": {
  "author_email": "AmirhosseinHeydari78@gmail.com",
  "author_github": "https://github.com/mr-pylin",
  "author_name": "Amirhossein Heydari",
  "kernelspec": {
   "display_name": "pytorch-workshop-U_zYfVTd-py3.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "origin_repo": "https://github.com/mr-pylin/pytorch-workshop"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}