simple_nn.py.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A simple neural network\n",
    "***\n",
    "\n",
    "The simple neural network consists of **two input, two hidden, and two output neurons**:\n",
    "![Alt text](drawing.svg)\n",
    "\n",
    "The input **(x_1, x_2)** are data points drawn from two different Gaussians with mean and covariance **((1, 1), diag(1 1)) and ((5, 5), diag(1 1))**, respectively.\n",
    "The output neurons **N_21 and N_22** define where the point is predicted to belong to class A or B, depending on which neuron has a higher output. The loss function is the squared error of the predictions, where the desired output is (1, 0), or (0, 1), depending on the class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib.pyplot import plot\n",
    "%matplotlib inline\n",
    "\n",
    "W = np.random.uniform(-0.1, 0.1, (2, 2, 3)) # Initialise weights of shape (2, 2, 3) - We have 2 2-by-3 Weight Matrices\n",
    "n = 0.01 # Learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we define the necessary functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def activation(x): # Assumes single 1D x\n",
    "    #return np.maximum(0, x)\n",
    "    x = np.clip(x, -500, 500)\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "def d_activation(x): # Assumes single 1D x\n",
    "    return activation(x)*(1-activation(x))\n",
    "def loss_fkt(x, y): # Assumes 2-D x, y\n",
    "    return np.sum(np.square(y-x))\n",
    "def d_loss(x, y): # Assumed 1-D x, y (for the respective neuron, as the other sum just equals 0)\n",
    "    return (y-x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can define a forward and a backward pass through the network. The backward pass returns the weight changes that are calculated with the backpropagation algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_pass(x):\n",
    "    x = np.array([x[0], x[1], 1])\n",
    "    sum_1 = np.dot(W[0], x)\n",
    "    ac_1 = np.array([activation(sum_1[0]), activation(sum_1[1]), 1])\n",
    "    sum_2 = np.dot(W[1], ac_1)\n",
    "    ac_2 = np.array([activation(sum_2[0]), activation(sum_2[1]), 1])\n",
    "    return sum_1, ac_1, sum_2, ac_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backward_pass(sum_1, ac_1, sum_2, ac_2, x):\n",
    "    label = x[2]\n",
    "    x = np.array([x[0], x[1], 1])\n",
    "    #1.) Weights W[1][0][:] - The weights on the connections to the first neuron of the second layer, N_21\n",
    "    y = 1 if label==1 else 0;\n",
    "    dw_1 = d_loss(ac_2[0], y)*d_activation(sum_2[0])\n",
    "    dw_1 = dw_1 * ac_1\n",
    "    #2.) Weights[1][1][:] for neutron N_22\n",
    "    y = 1 if label==-1 else 0;\n",
    "    dw_2 = d_loss(ac_2[1], y)*d_activation(sum_2[1])\n",
    "    dw_2 = dw_2 * ac_1\n",
    "    \n",
    "    #3) Weights[0][0][:] for Neuron N_11\n",
    "    # First path over N_21\n",
    "    y = 1 if label==1 else 0;\n",
    "    dw_31 = d_loss(ac_2[0], y)*d_activation(sum_2[0])*W[1][0][0]*d_activation(sum_1[0])\n",
    "    dw_31 = dw_31 * x\n",
    "    # Second path over N_22\n",
    "    y = 1 if label==-1 else 0;\n",
    "    dw_32 = d_loss(ac_2[1], y)*d_activation(sum_2[1])*W[1][1][0]*d_activation(sum_1[0])\n",
    "    dw_32 = dw_32 * x\n",
    "    #Sum of both\n",
    "    dw_3 = dw_31 + dw_32\n",
    "    \n",
    "    #4) Weights[0][1][:] for Neuron N_12\n",
    "    # First path over N_21\n",
    "    y = 1 if label==1 else 0;\n",
    "    dw_41 = d_loss(ac_2[0], y)*d_activation(sum_2[0])*W[1][0][1]*d_activation(sum_1[1])\n",
    "    dw_41 = dw_41 * x\n",
    "    # Second path over N_22\n",
    "    y = 1 if label==-1 else 0;\n",
    "    dw_42 = d_loss(ac_2[1], y)*d_activation(sum_2[1])*W[1][1][1]*d_activation(sum_1[1])\n",
    "    dw_42 = dw_42 * x\n",
    "    #Sum of both\n",
    "    dw_4 = dw_41 + dw_42\n",
    "    \n",
    "    #Full change matrix\n",
    "    dw = np.array([[dw_3, dw_4], [dw_1, dw_2]])\n",
    "    return dw\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_weights(dw):\n",
    "    global W\n",
    "    W -= n*dw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can produce some data. The data is collected from 2 2-D Gaussians with mean and covariance ((1, 1), diag(1 1)) and ((5, 5), diag(1 1))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data = np.concatenate([np.random.randn(1000, 2), np.random.randn(1000, 2)+5])\n",
    "#plot(data[:, 0], data[:, 1], 'bo')\n",
    "num_data_points = 1000\n",
    "data_g1 = np.hstack((np.random.randn(num_data_points, 2), np.ones(num_data_points).reshape(-1, 1)))\n",
    "data_g2 = np.hstack((np.random.randn(num_data_points, 2)+5, -1*np.ones(num_data_points).reshape(-1, 1)))\n",
    "data = np.concatenate([data_g1, data_g2])\n",
    "np.random.shuffle(data)\n",
    "\n",
    "# Split data into training and validation set\n",
    "ind = int(num_data_points*2*0.75)\n",
    "data_train = data[:ind][:]\n",
    "data_eval = data[ind:][:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "***\n",
    "We can now run through all data points 50 times and update the weights accordingly. As we perform an online update, we implement the **stochastic gradient descent** algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(50):\n",
    "    for x in data_train:\n",
    "        sum_1, ac_1, sum_2, ac_2 = forward_pass([x[0], x[1]])\n",
    "        dw = backward_pass(sum_1, ac_1, sum_2, ac_2, x)\n",
    "        update_weights(dw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validation\n",
    "***\n",
    "Finally, we can check our results and have a look at the performance accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy 1.0\n"
     ]
    }
   ],
   "source": [
    "correct_assignments = 0\n",
    "for x in data_eval:\n",
    "    _, _, _, ac_2 = forward_pass([x[0], x[1]])\n",
    "    y_hat = 1 if ac_2[0]<ac_2[1] else -1;\n",
    "    if y_hat==x[2]:\n",
    "        correct_assignments += 1\n",
    "acc = correct_assignments*1.0/len(data_eval)\n",
    "print (\"accuracy\", acc)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}