code/utils/custom-classes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📝 **Author:** Amirhossein Heydari - 📧 **Email:** <amirhosseinheydari78@gmail.com> - 📍 **Origin:** [mr-pylin/pytorch-workshop](https://github.com/mr-pylin/pytorch-workshop)\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- [Dependencies](#toc1_)    \n",
    "- [Custom Classes in PyTorch](#toc2_)    \n",
    "  - [Load Breast Cancer Wisconsin (Diagnostic) Dataset](#toc2_1_)    \n",
    "  - [Custom Dataset](#toc2_2_)    \n",
    "  - [Custom Transform](#toc2_3_)    \n",
    "    - [Direct transform](#toc2_3_1_)    \n",
    "    - [Integrated transform](#toc2_3_2_)    \n",
    "  - [Custom Activation Function](#toc2_4_)    \n",
    "  - [Custom Model](#toc2_5_)    \n",
    "  - [Custom Loss Function](#toc2_6_)    \n",
    "  - [Custom Optimizer](#toc2_7_)    \n",
    "- [Example: All In One](#toc3_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=1\n",
    "\tmaxLevel=6\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc1_'></a>[Dependencies](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import torch\n",
    "from torch import nn, optim\n",
    "from torch.utils.data import DataLoader, Dataset, TensorDataset\n",
    "from torchvision.transforms import v2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc2_'></a>[Custom Classes in PyTorch](#toc0_)\n",
    "\n",
    "- **PyTorch** is a **flexible** deep learning framework that allows developers to **customize** different components according to their **specific needs**.\n",
    "- This flexibility is essential for **implementing** custom **datasets**, **models**, and **optimization** routines, which may not be covered by the **built-in classes**.\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_1_'></a>[Load Breast Cancer Wisconsin (Diagnostic) Dataset](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "breast_cancer_dataset_url = (\n",
    "    r\"https://github.com/mr-pylin/datasets/raw/refs/heads/main/data/tabular-data/breast-cancer-wisconsin-diagnostic/dataset.csv\"\n",
    ")\n",
    "\n",
    "# pandas data-frame\n",
    "df = pd.read_csv(breast_cancer_dataset_url, encoding=\"utf-8\")\n",
    "\n",
    "# encode labels\n",
    "df[\"Diagnosis\"] = df[\"Diagnosis\"].map({\"B\": 0, \"M\": 1})\n",
    "\n",
    "# log\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = df[\"Diagnosis\"].unique()\n",
    "class_to_idx = {l: i for i, l in enumerate(classes)}\n",
    "\n",
    "# split dataset into features and labels\n",
    "X, y = df.iloc[:, 2:].values, df.iloc[:, 1].values\n",
    "\n",
    "# convert categorical labels into indices\n",
    "y = np.array([class_to_idx[l] for l in y])\n",
    "\n",
    "# properties of the dataset\n",
    "num_samples, num_features = X.shape\n",
    "classes, samples_per_class = np.unique(y, return_counts=True)\n",
    "\n",
    "# log\n",
    "print(f\"X.shape: {X.shape}\")\n",
    "print(f\"X.dtype: {X.dtype}\")\n",
    "print(f\"y.shape: {y.shape}\")\n",
    "print(f\"y.dtype: {y.dtype}\")\n",
    "print(\"-\" * 50)\n",
    "print(f\"classes          : {classes}\")\n",
    "print(f\"samples per class: {samples_per_class}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_2_'></a>[Custom Dataset](#toc0_)\n",
    "\n",
    "- PyTorch’s `Dataset` class can be easily subclassed to define custom datasets\n",
    "- This allows you to load and preprocess your data according to your needs.\n",
    "- Use `torch.utils.data.Dataset` as the parent class and override `__len__` and `__getitem__`.\n",
    "\n",
    "📝 **Docs & Tutorials** 📚:\n",
    "\n",
    "- Data Loading Utility: [pytorch.org/docs/stable/data.html](https://pytorch.org/docs/stable/data.html)\n",
    "- Datasets & DataLoaders: [pytorch.org/tutorials/beginner/basics/data_tutorial.html](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert numpy.ndarray to torch.Tensor\n",
    "X = torch.tensor(X, dtype=torch.float32)\n",
    "y = torch.tensor(y, dtype=torch.float32).view(-1, 1)\n",
    "\n",
    "# log\n",
    "print(f\"type(X): {type(X)}  |  X.dtype: {X.dtype}  |  X.shape: {X.shape}\")\n",
    "print(f\"type(y): {type(y)}  |  y.dtype: {y.dtype}  |  y.shape: {y.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomDataset(Dataset):\n",
    "    def __init__(self, data: torch.Tensor, labels: torch.Tensor):\n",
    "        self.data = data\n",
    "        self.labels = labels\n",
    "\n",
    "    def __len__(self) -> int:\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, index: int) -> tuple[torch.Tensor, torch.Tensor]:\n",
    "        return self.data[index], self.labels[index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a pytorch dataset\n",
    "dataset_1 = CustomDataset(X, y)  # custom\n",
    "dataset_2 = TensorDataset(X, y)  # built-in\n",
    "\n",
    "# log\n",
    "print(f\"type(dataset_1) : {type(dataset_1)}\")\n",
    "print(f\"len(dataset_1)  : {len(dataset_1)}\")\n",
    "print(f\"dataset_1[0]    : {dataset_1[0]}\")\n",
    "print(\"-\" * 50)\n",
    "print(f\"type(dataset_2) : {type(dataset_2)}\")\n",
    "print(f\"len(dataset_2)  : {len(dataset_2)}\")\n",
    "print(f\"dataset_2[0]    : {dataset_2[0]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_3_'></a>[Custom Transform](#toc0_)\n",
    "\n",
    "- Transforms are used to modify the input data before feeding it into the model.\n",
    "- PyTorch provides a lot of built-in transforms (like cropping, flipping, etc.) in `torchvision.transforms`.\n",
    "- you can define your own transformation by implementing the `__call__` method.\n",
    "\n",
    "📝 **Docs & Tutorials** 📚:\n",
    "\n",
    "- Transforming and augmenting images: [pytorch.org/vision/stable/transforms.html](https://pytorch.org/vision/stable/transforms.html)\n",
    "- Transforms: [pytorch.org/tutorials/beginner/basics/transforms_tutorial.html](https://pytorch.org/tutorials/beginner/basics/transforms_tutorial.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NumpyToTensor:\n",
    "    def __call__(self, sample: np.ndarray) -> tuple[torch.Tensor, torch.Tensor]:\n",
    "        converted_sample = torch.tensor(sample[0], dtype=torch.float32), torch.tensor(sample[1], dtype=torch.float32)\n",
    "        return converted_sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class NormalizeTo01:\n",
    "    def __init__(self):\n",
    "        self.min_val = None\n",
    "        self.max_val = None\n",
    "\n",
    "    def fit(self, data: np.ndarray) -> None:\n",
    "        self.min_val = np.min(data, axis=0).astype(np.float32)\n",
    "        self.max_val = np.max(data, axis=0).astype(np.float32)\n",
    "\n",
    "    def __call__(self, sample: tuple[torch.Tensor, torch.Tensor]) -> tuple[torch.Tensor, torch.Tensor]:\n",
    "        normalized_sample = (sample[0] - self.min_val) / (self.max_val - self.min_val), sample[1]\n",
    "        return normalized_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc2_3_1_'></a>[Direct transform](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# dataset\n",
    "X = np.array([[1, 2, 3], [5, 1, 2], [3, 3, 3]])\n",
    "y = np.array([[0], [0], [1]])\n",
    "dataset = list(zip(X, y))\n",
    "\n",
    "# log\n",
    "print(f\"dataset : {dataset}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NumpyToTensor\n",
    "t_totensor = NumpyToTensor()\n",
    "\n",
    "# NormalizeTo01\n",
    "t_normalize = NormalizeTo01()\n",
    "t_normalize.fit(X)\n",
    "\n",
    "# transform the first input\n",
    "result = t_normalize(t_totensor(dataset[0]))\n",
    "\n",
    "# log\n",
    "print(f\"result          : {result}\")\n",
    "print(f\"result[0].dtype : {result[0].dtype}\")\n",
    "print(f\"result[1].dtype : {result[1].dtype}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a id='toc2_3_2_'></a>[Integrated transform](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# advanced Dataset with transform support\n",
    "class AdvancedCustomDataset(Dataset):\n",
    "    def __init__(self, data: torch.Tensor, labels: torch.Tensor, transform: v2.Compose | None = None):\n",
    "        self.data = data\n",
    "        self.labels = labels\n",
    "        self.transform = transform\n",
    "\n",
    "        # if NormalizeTo01 is included, call <fit> method for that\n",
    "        if self.transform:\n",
    "            for t in self.transform.transforms:\n",
    "                if isinstance(t, NormalizeTo01):\n",
    "                    t.fit(self.data)\n",
    "                    break\n",
    "\n",
    "    def __len__(self) -> int:\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, index: int) -> tuple[torch.Tensor, torch.Tensor]:\n",
    "        sample = self.data[index], self.labels[index]\n",
    "\n",
    "        if self.transform:\n",
    "            sample = self.transform(sample)\n",
    "\n",
    "        return sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = np.array([[1, 2, 3], [5, 1, 2], [3, 3, 3]])\n",
    "y = np.array([[0], [0], [1]])\n",
    "\n",
    "transformations = v2.Compose(\n",
    "    [\n",
    "        NumpyToTensor(),\n",
    "        NormalizeTo01(),\n",
    "    ]\n",
    ")\n",
    "\n",
    "dataset = AdvancedCustomDataset(X, y, transformations)\n",
    "\n",
    "# log\n",
    "for i in range(len(y)):\n",
    "    print(f\"dataset[{i}]: {dataset[i]}\")\n",
    "    print(f\"    -> input data : {dataset[i][0]}\")\n",
    "    print(f\"    -> label      : {dataset[i][1]}\", end=\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_4_'></a>[Custom Activation Function](#toc0_)\n",
    "\n",
    "- you can create your own activation function by subclassing `torch.nn.Module`.\n",
    "- Use `torch.nn.Module` as the parent class and implement  `forward` method.\n",
    "\n",
    "📝 **Docs & Tutorials** 📚:\n",
    "\n",
    "- Non-linear Activations (weighted sum, nonlinearity): [pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity)\n",
    "- Non-linear Activations (other): [pytorch.org/docs/stable/nn.html#non-linear-activations-other](https://pytorch.org/docs/stable/nn.html#non-linear-activations-other)\n",
    "- Non-linear activation functions: [pytorch.org/docs/stable/nn.functional.html#non-linear-activation-functions](https://pytorch.org/docs/stable/nn.functional.html#non-linear-activation-functions)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomSigmoid(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        return torch.sigmoid(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sig_1 = CustomSigmoid()\n",
    "sig_2 = nn.Sigmoid()\n",
    "\n",
    "values = torch.tensor([10, 0, -10], dtype=torch.float32)\n",
    "\n",
    "# log\n",
    "print(f\"sig_1(values) : {sig_1(values)}\")\n",
    "print(f\"sig_2(values) : {sig_2(values)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_5_'></a>[Custom Model](#toc0_)\n",
    "\n",
    "- Sequential Model:\n",
    "  - Useful for simpler models where the layers are stacked in a linear sequence\n",
    "  - The `torch.nn.Sequential` class allows you to stack layers in a sequence, passing the output of one layer directly to the next.\n",
    "  - This is great for simple models like fully-connected neural networks or basic CNNs.\n",
    "  - Key Points\n",
    "    - Layers are defined in the order they are passed to `Sequential`.\n",
    "    - You don't need to define the `forward` method manually; PyTorch handles it for you.\n",
    "- Functional Model:\n",
    "  - Allowing for complex architectures where you might need non-linear layer connections (e.g., skip connections in ResNet)\n",
    "  - models are created by subclassing `torch.nn.Module`.\n",
    "  - This allows you to define any neural network architecture, from simple feedforward networks to complex architectures like GANs or transformers\n",
    "  - Key Points\n",
    "    - Use `torch.nn.Module` as the parent class and implement  `forward` method.\n",
    "\n",
    "📝 **Docs & Tutorials** 📚:\n",
    "\n",
    "- Module: [pytorch.org/docs/stable/generated/torch.nn.Module.html](https://pytorch.org/docs/stable/generated/torch.nn.Module.html)\n",
    "- torch.nn: [pytorch.org/docs/stable/nn.html](https://pytorch.org/docs/stable/nn.html)\n",
    "- Building Models with PyTorch: [pytorch.org/tutorials/beginner/introyt/modelsyt_tutorial.html](https://pytorch.org/tutorials/beginner/introyt/modelsyt_tutorial.html)\n",
    "- Build the Neural Network: [pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html](https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html)\n",
    "- Neural Networks: [pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial](https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a simple sequential model\n",
    "model_1 = nn.Sequential(\n",
    "    nn.Linear(in_features=30, out_features=16),\n",
    "    nn.Sigmoid(),\n",
    "    nn.Linear(in_features=16, out_features=1),\n",
    "    nn.Sigmoid(),\n",
    ")\n",
    "\n",
    "# log\n",
    "model_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a simple functional model\n",
    "class CustomLogisticRegression(nn.Module):\n",
    "    def __init__(self, input_size: int, hidden_size: int, output_size: int):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(in_features=input_size, out_features=hidden_size)\n",
    "        self.sigmoid1 = CustomSigmoid()\n",
    "        self.fc2 = nn.Linear(in_features=hidden_size, out_features=output_size)\n",
    "        self.sigmoid2 = CustomSigmoid()\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = self.fc1(x)\n",
    "        x = self.sigmoid1(x)\n",
    "        x = self.fc2(x)\n",
    "        x = self.sigmoid2(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "# initialize the model\n",
    "model_2 = CustomLogisticRegression(30, 16, 1)\n",
    "\n",
    "# log\n",
    "model_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_6_'></a>[Custom Loss Function](#toc0_)\n",
    "\n",
    "- PyTorch comes with standard loss functions like MSE, Cross-Entropy, etc.\n",
    "- you can create your own loss function by subclassing `torch.nn.Module`.\n",
    "- Use `torch.nn.Module` as the parent class and implement  `forward` method.\n",
    "\n",
    "📝 **Docs & Tutorials** 📚:\n",
    "\n",
    "- Loss Functions: [pytorch.org/docs/stable/nn.html#loss-functions](https://pytorch.org/docs/stable/nn.html#loss-functions)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomMSE(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, y_pred: torch.Tensor, y_true: torch.Tensor) -> torch.Tensor:\n",
    "        loss = torch.mean((y_pred - y_true) ** 2)\n",
    "        return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split dataset into (data, labels)\n",
    "X = dataset_1[:][0]\n",
    "y_true = dataset_1[:][1]\n",
    "\n",
    "# feed-forward\n",
    "y_pred = model_1(X)\n",
    "\n",
    "# MSE loss function\n",
    "criterion_1 = CustomMSE()  # custom\n",
    "criterion_2 = nn.MSELoss()  # built-in\n",
    "\n",
    "# compute the loss\n",
    "loss_1 = criterion_1(y_pred, y_true)\n",
    "loss_2 = criterion_2(y_pred, y_true)\n",
    "\n",
    "# log\n",
    "print(f\"loss_1: {loss_1}\")\n",
    "print(f\"loss_2: {loss_2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc2_7_'></a>[Custom Optimizer](#toc0_)\n",
    "\n",
    "- PyTorch offers optimizers like SGD, Adam, etc.\n",
    "- you can create your own optimizer by subclassing `torch.optim.Optimizer`.\n",
    "- Use `torch.optim.Optimizer` as the parent class and override `step` method.\n",
    "\n",
    "📝 **Docs & Tutorials** 📚:\n",
    "\n",
    "- torch.optim: [pytorch.org/docs/stable/optim.html](https://pytorch.org/docs/stable/optim.html)\n",
    "- torch.optim.Optimizer.step: [pytorch.org/docs/stable/generated/torch.optim.Optimizer.step.html](https://pytorch.org/docs/stable/generated/torch.optim.Optimizer.step.html)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this implementation might not be the same as SGD\n",
    "class CustomSGD(optim.Optimizer):\n",
    "    def __init__(self, params, lr: float = 0.01, momentum: float = 0):\n",
    "        defaults = dict(lr=lr, momentum=momentum)\n",
    "        super().__init__(params, defaults)\n",
    "\n",
    "    def step(self) -> None:\n",
    "        for group in self.param_groups:\n",
    "            lr = group[\"lr\"]\n",
    "            momentum = group[\"momentum\"]\n",
    "\n",
    "            for p in group[\"params\"]:\n",
    "                if p.grad is None:\n",
    "                    continue\n",
    "                d_p = p.grad.data\n",
    "\n",
    "                if momentum != 0:\n",
    "                    param_state = self.state[p]\n",
    "                    if \"momentum_buffer\" not in param_state:\n",
    "                        buf = param_state[\"momentum_buffer\"] = torch.zeros_like(p.data)\n",
    "                        buf.mul_(momentum).add_(d_p)\n",
    "                    else:\n",
    "                        buf = param_state[\"momentum_buffer\"]\n",
    "                        buf.mul_(momentum).add_(d_p, alpha=1 - momentum)\n",
    "                    d_p = buf\n",
    "\n",
    "                p.data.add_(d_p, alpha=-lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer_1 = CustomSGD(model_1.parameters())\n",
    "optimizer_2 = optim.SGD(model_1.parameters())\n",
    "\n",
    "# log\n",
    "print(f\"optimizer_1:\\n{optimizer_1}\\n\")\n",
    "print(f\"optimizer_2:\\n{optimizer_2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc3_'></a>[Example: All In One](#toc0_)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load breast-cancer dataset\n",
    "X, y = df.iloc[:, 2:].values, df.iloc[:, 1].values\n",
    "\n",
    "\n",
    "# create a custom Dataset class\n",
    "class CustomDataset(Dataset):\n",
    "    def __init__(self, data: torch.Tensor, labels: torch.Tensor):\n",
    "        self.data = data\n",
    "        self.labels = labels\n",
    "\n",
    "    def __len__(self) -> int:\n",
    "        return len(self.data)\n",
    "\n",
    "    def __getitem__(self, index: int) -> tuple[torch.Tensor, torch.Tensor]:\n",
    "        return self.data[index], self.labels[index]\n",
    "\n",
    "\n",
    "# convert numpy.ndarray to torch.Tensor\n",
    "X = torch.tensor(X, dtype=torch.float32)\n",
    "y = torch.tensor(y, dtype=torch.float32).view(-1, 1)\n",
    "\n",
    "# create a dataset\n",
    "dataset = CustomDataset(X, y)\n",
    "\n",
    "# create a dataloader\n",
    "dataloader = DataLoader(dataset, batch_size=8, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# custom sigmoid activation\n",
    "class CustomSigmoid(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        return torch.sigmoid(x)\n",
    "\n",
    "\n",
    "# model\n",
    "class CustomLogisticRegression(nn.Module):\n",
    "    def __init__(self, input_size: int, hidden_size: int, output_size: int):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(in_features=input_size, out_features=hidden_size)\n",
    "        self.sigmoid1 = CustomSigmoid()\n",
    "        self.fc2 = nn.Linear(in_features=hidden_size, out_features=output_size)\n",
    "        self.sigmoid2 = CustomSigmoid()\n",
    "\n",
    "    def forward(self, x: torch.Tensor) -> torch.Tensor:\n",
    "        x = self.fc1(x)\n",
    "        x = self.sigmoid1(x)\n",
    "        x = self.fc2(x)\n",
    "        x = self.sigmoid2(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "input_size = X.shape[1]\n",
    "hidden_size = 2\n",
    "output_size = y.shape[1]\n",
    "\n",
    "# model\n",
    "model = CustomLogisticRegression(input_size, hidden_size, output_size)\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyper parameters\n",
    "epochs = 10\n",
    "lr = 0.005\n",
    "criterion = CustomMSE()\n",
    "optimizer = CustomSGD(model.parameters(), lr=lr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# training loop\n",
    "model.train()\n",
    "total_loss = []\n",
    "total_acc = []\n",
    "\n",
    "for epoch in range(epochs):\n",
    "\n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "\n",
    "    for x, y_true in dataloader:\n",
    "\n",
    "        # forward\n",
    "        y_pred = model(x)\n",
    "        loss = criterion(y_pred, y_true)\n",
    "\n",
    "        # backward\n",
    "        loss.backward()\n",
    "\n",
    "        # update parameters\n",
    "        optimizer.step()\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # store loss and accuracy per iteration\n",
    "        epoch_loss += loss.item()\n",
    "        epoch_acc += ((y_pred > 0.5).float() == y_true).sum().item()\n",
    "\n",
    "    total_loss.append(epoch_loss / len(dataloader))\n",
    "    total_acc.append(epoch_acc / len(X))\n",
    "\n",
    "    # log\n",
    "    print(f\"epoch {epoch+1:0{len(str(epochs))}}/{epochs} -> loss: {total_loss[-1]:.5f} - accuracy: {total_acc[-1]*100:.2f}%\")"
   ]
  }
 ],
 "metadata": {
  "author_email": "AmirhosseinHeydari78@gmail.com",
  "author_github": "https://github.com/mr-pylin",
  "author_name": "Amirhossein Heydari",
  "kernelspec": {
   "display_name": "pytorch-workshop-U_zYfVTd-py3.12",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  },
  "origin_repo": "https://github.com/mr-pylin/pytorch-workshop"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}