01_basic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basics\n",
    "In this tutorial we go over the basic process of constructing and training a simple neural network\n",
    "in DeepFusion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "In DeepFusion, all networks are composed by combining 3 basic types of `components`:\n",
    "- `Data`\n",
    "- `Module`\n",
    "- `Net`\n",
    "Therefore, we need to import the components needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Import Net, Data and necessary modules\n",
    "from deepfusion.components.net import Net\n",
    "from deepfusion.components.data import Data\n",
    "from deepfusion.components.modules import MatMul\n",
    "from deepfusion.components.modules.activation_functions import Relu\n",
    "from deepfusion.components.modules.loss_functions import MSELoss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(linewidth = np.inf)\n",
    "# Set seed for reproducibility\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Training Data\n",
    "We will use the following function as a running example in the demo scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(X):\n",
    "    # For a training example - y = x1 + 2 * x2^2 + 3 * x3^.5, the fancy indexing preserves 2D shape\n",
    "    Y = X[:, 0:1] + 2 * X[:, 1:2] ** 2 + 3 * X[:, 2:3] ** 0.5\n",
    "    return Y\n",
    "\n",
    "# We will use m for number of examples\n",
    "m_train = 100000\n",
    "factor = 5\n",
    "X_train = np.random.rand(m_train, 3) * factor\n",
    "Y_train = f(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Construct Neural Network\n",
    "We construct the following neural network architecture:\n",
    "\n",
    "`x -> Matmul -> z1 -> Relu -> a -> Matmul -> z2, + y -> MSE -> loss`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = Data(ID = 'x', shape = (1, 3))\n",
    "\n",
    "z1 = Data(ID = 'z1', shape = (1, 5))\n",
    "Matmul1 = MatMul(ID = 'Matmul1', inputs = [x], output = z1)\n",
    "\n",
    "a = Data(ID = 'a', shape = (1, 5))\n",
    "ActF = Relu(ID = 'ActF', inputs = [z1], output = a)\n",
    "\n",
    "z2 = Data(ID = 'z2', shape = (1, 1))\n",
    "Matmul2 = MatMul(ID = 'Matmul2', inputs = [a], output = z2)\n",
    "\n",
    "# Add target variable, loss variable and loss function\n",
    "y = Data('y', shape = (1, 1))\n",
    "loss = Data('loss', shape = (1, 1))\n",
    "LossF = MSELoss(ID = 'LossF', inputs = [z2, y], output = loss)\n",
    "\n",
    "# Initialize the neural network\n",
    "net = Net(ID = 'Net', root_nodes = [loss])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To keep this script simple we don't define an optimizer, we see this in the next demo.\n",
    "\n",
    "By default the Adam optimizer will be used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Neural Network\n",
    "We need to first set the training properties and then go through the following basic procedure:\n",
    "1. Setting the `val` attribute of the input nodes.\n",
    "2. Running the `forward()` pass.\n",
    "3. Running the `backward()` pass.\n",
    "4. Performing a network `update()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: [1/1000]. Cost: 512.1406868659374.\n",
      "Epoch: [100/1000]. Cost: 435.3471895342655.\n",
      "Epoch: [200/1000]. Cost: 409.49528139407636.\n",
      "Epoch: [300/1000]. Cost: 217.10333840630034.\n",
      "Epoch: [400/1000]. Cost: 211.5108474897141.\n",
      "Epoch: [500/1000]. Cost: 117.1519258020598.\n",
      "Epoch: [600/1000]. Cost: 129.90904088681503.\n",
      "Epoch: [700/1000]. Cost: 91.49154702905682.\n",
      "Epoch: [800/1000]. Cost: 72.32839497910018.\n",
      "Epoch: [900/1000]. Cost: 54.38000727140997.\n",
      "Epoch: [1000/1000]. Cost: 37.23287852578279.\n"
     ]
    }
   ],
   "source": [
    "epochs = 1000\n",
    "print_cost_every = 100\n",
    "learning_rate = 0.001\n",
    "B = 64\n",
    "\n",
    "net.set_learning_rate(learning_rate)\n",
    "\n",
    "for epoch in range(1, epochs + 1):\n",
    "    # Get a mini-batch of the data\n",
    "    idx = np.random.choice(X_train.shape[0], size = B, replace = False)\n",
    "    # Feed in the data to the input nodes of the network for forward pass to be performed\n",
    "    x.val = X_train[idx, :]\n",
    "    y.val = Y_train[idx, :]\n",
    "\n",
    "    # Run forward pass\n",
    "    net.forward()\n",
    "\n",
    "    # Run backward pass\n",
    "    net.backward()\n",
    "\n",
    "    # Update the network parameters\n",
    "    net.update()\n",
    "\n",
    "    # Print the cost during training every few iterations\n",
    "    if epoch % print_cost_every == 0 or epoch == 1:\n",
    "        J = loss.val\n",
    "        print(f'Epoch: [{epoch}/{epochs}]. Cost: {J}.')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}