linear_regression_exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> linear Regression </h1>\n",
    "\n",
    "Welcome to the 2nd day of the workshop. Today we are going to implement linear regression from scratch. In the first workshop you learned the use of numpy, pandas and matplotlib, we are going to use all these libraries for developing our model.\n",
    "There are mainly two classical problems in supervised machine learning: 1) Regression problems 2) Classification problems\n",
    "\n",
    "Regression Problem:- A regression problem is when the output variable is a real or continuous value, such as “salary” or “weight”. Many different models can be used, the simplest is the linear regression. It tries to fit data with the best hyper-plane which goes through the points. So when ever you see real and continuous output variable understand it is a regression problem.\n",
    "\n",
    "Classification Problem :- A classification problem is when the output variable is a category, such as “red” or “blue” or “disease” and “no disease”. A classification model attempts to draw some conclusion from observed values. Given one or more inputs a classification model will try to predict the value of one or more outcomes. For example, when filtering emails “spam” or “not spam”, when looking at transaction data, “fraudulent”, or “authorized”. In short Classification either predicts categorical class labels or classifies data (construct a model) based on the training set and the values (class labels) in classifying attributes and uses it in classifying new data.\n",
    "\n",
    "So today we are going to implement linear regression from scratch for a regression problem, so let's get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#loading all the necessary libraries\n",
    "import numpy as np\n",
    "from sklearn.datasets import load_boston\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to use boston housing price dataset for a given task, it is a classical dataset for a regression task, It contains around 506 data entries containing 13 different features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_boston()\n",
    "\n",
    "print(dataset.DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = dataset.data\n",
    "y = dataset.target\n",
    "\n",
    "y = np.expand_dims(y,axis=1)\n",
    "\n",
    "print(\"total number of samples in dataset is: {}\".format(x.shape[0]))\n",
    "print(\"total features in dataset is: {}\".format(x.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most important task in data science and machine learning is the cleaning of data, lucky for us this dataset is alreay cleaned so we can focus on visulization and modelling part. Data visulization is very necessary to understand dependancies of target values to the feature values and also gain some insights regarding what might be the best features to choose to fit the machine learning model. This dataset is prepared containing useful features only so we are going to use every features for now, but anyways visulization is good habit in data science."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#basic visulization\n",
    "plt.figure(figsize=(5, 4))\n",
    "plt.hist(dataset.target)\n",
    "plt.title('Boston Housing Prices and Count Histogram')\n",
    "plt.xlabel('price ($1000s)')\n",
    "plt.ylabel('count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plotting every feature relation with target\n",
    "for index, feature_name in enumerate(dataset.feature_names):\n",
    "    plt.figure(figsize=(5,4))\n",
    "    plt.tight_layout()\n",
    "    plt.scatter(dataset.data[:, index], dataset.target)\n",
    "    plt.ylabel('Price',size = 12)\n",
    "    plt.xlabel(feature_name, size = 12)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Linear Model</h1>\n",
    "\n",
    "The idea of linear regression is to fit a line to a set of points. So let's use the line function given by:\n",
    "f(x)=y=mx+b\n",
    " \n",
    "where m is the slope and b is our y intercept, or for a more general form (multiple variables)\n",
    "h(x)=θ0x0+θ1x1+θ2x2+...+θnxn\n",
    " \n",
    "such that for a single variable where n = 1,\n",
    "h(x)=θ0+θ1x1\n",
    " \n",
    "when x0=1\n",
    " \n",
    "where theta is our parameters (slope and intercept) and h(x) is our hypothesis or predicted value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Magic of Matices</h1>\n",
    "\n",
    "Also we are going to do computation in vectorized form not iterative for, so let's just brush up the matrix concepts:\n",
    "\n",
    "Suppose A is our feature matrix X and B as our parameter matrix theta, that is, \n",
    "    X=[ 1 2 ] θ=[ 2 3 ]\n",
    "      [ 1 3 ]\n",
    "      [ 1 4 ]\n",
    "\n",
    "Remember that we have our linear model h(x)=θ0x0+θ1x1\n",
    "\n",
    "We know that X0=[ 1 ] X1=[ 2 ] θT=[ 2 ]\n",
    "                \n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&nbsp;[ 1 ]&emsp;&emsp;[ 3 ]&emsp;&emsp;[ 3 ]\n",
    "                \n",
    "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&nbsp;[ 1 ]&emsp;&emsp;[ 4 ]\n",
    "\n",
    "then we can actually use matrix dot product to do the multiplication and addition at the same time (and faster)\n",
    "\n",
    "H=[ θ0X00+θ1X01 ]=[ θ0+θ1X01 ]=[ 2+3(2) ]=[ 8 ]\n",
    "\n",
    "  &emsp;&nbsp;[ θ0X10+θ1X11 ]&ensp;&nbsp;[ θ0+θ1X11 ]&ensp;[ 2+3(3) ]&ensp;[ 11 ]\n",
    "\n",
    "  &emsp;&nbsp;[ θ0X20+θ1X21 ]&ensp;&nbsp;[ θ0+θ1X21 ]&ensp;[ 2+3(4) ]&ensp;[ 14 ]\n",
    "\n",
    "can be as simple as\n",
    "\n",
    "H=X dot θ\n",
    "\n",
    "Yes, that is the power of Matrices!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#initializing the weights\n",
    "#Task :- initialize vector of zeros according for x dot w remember size of x is (506,13) \n",
    "#and one bias vector of one will be add so size will be (506,14) so initialize vector accordingly\n",
    "\n",
    "# CODE HERE\n",
    "\n",
    "w = # approximately one line  Hint: use np.ones\n",
    "print(w.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Importance of Normalization </h1>\n",
    "\n",
    "Normalization is very much important, consider a case: house price depends on area of house and rooms in the house, we know that area of houses is in range of 900-100 sqft. while number of bedrooms are at most 5-6, so obviously target values should depend more on house area but that's not the case, so we have to normalize data by mean and standard deviation to get every data points in comparable range. \n",
    "\n",
    "Normalization avoids these problems by creating new values that maintain the general distribution and ratios in the source data, while keeping values within a scale applied across all numeric columns used in the model.\n",
    "\n",
    "Also normalization makes convergence process fast."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize(x,mode='std'):\n",
    "    n_samples = # use x.shape and get number of samples\n",
    "    #code here\n",
    "    \n",
    "    x_mean = # find mean of every feature column Hint: use np.mean\n",
    "    if mode == 'std':\n",
    "        x_std = #find standard deviation of every feature column Hint: use np.std\n",
    "    else:\n",
    "        x_std = #find maximum of every column Hint : use np.max\n",
    "        \n",
    "    x = # subtract x_mean from x and divide by x_std \n",
    "    \n",
    "    x = np.hstack((np.ones((n_samples,1)),x))   # stacking bias (vector of 1(constant)) to x\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)   # do not change seed otherwise answer will not match  \n",
    "x = np.random.rand(5,5)\n",
    "x = normalize(x,mode = 'std')\n",
    "print(x.shape)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Answer should match below </h3>\n",
    "(5, 6)\n",
    "\n",
    "\n",
    "[[ 1.         -0.20602098  0.78951027 -0.41884901 -1.72218407  0.17681922]\n",
    " [ 1.          0.11854592 -0.22998031  1.34645231  1.12080062  0.03565705]\n",
    " [ 1.          0.6060984   0.10534616 -0.6309152   0.86237971 -1.06099139]\n",
    " [ 1.         -1.74955693 -1.76276226  0.98513859 -0.13855101  1.74368392]\n",
    " [ 1.          1.23093358  1.09788614 -1.28182669 -0.12244524 -0.8951688 ]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> Cost Function </h1>\n",
    "\n",
    "It is a function that measures the performance of a Machine Learning model for given data. Cost Function quantifies the error between predicted values and expected values and presents it in the form of a single real number. Depending on the problem Cost Function can be formed in many different ways. The purpose of Cost Function is to be either:\n",
    "\n",
    "Minimized - then returned value is usually called cost, loss or error. The goal is to find the values of model parameters for which Cost Function return as small number as possible.\n",
    "\n",
    "Maximized - then the value it yields is named a reward. The goal is to find values of model parameters for which returned number is as large as possible.\n",
    "\n",
    "For algorithms relying on Gradient Descent to optimize model parameters, every function has to be differentiable.\n",
    "\n",
    "We are going to use mean squared error as a cost function here.\n",
    "\n",
    "<h1> Mean squared error </h1>\n",
    "\n",
    "Regression metric which measures the average magnitude of errors in a group of predictions, without considering their directions. In other words, it’s a mean of absolute differences among predictions and expected results where all individual deviations have even importance.\n",
    "\n",
    "  <h3>MSE&nbsp;=&nbsp;1/N&nbsp;∑<sub>i=1</sub><sup>n</sup>(yi−(mxi+b))<sup>2</sup> </h3>\n",
    "                        \n",
    "where:\n",
    "i - index of sample,\n",
    "ŷ - predicted value,\n",
    "y - expected value,\n",
    "m - number of samples in dataset.\n",
    "\n",
    "Sometimes it is possible to see the form of formula with swapped predicted value and expected value, but it works the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_cost(x,y,w):  \n",
    "    #CODE HERE\n",
    "    \n",
    "    no_samples = #Get the value of number of samples from x \n",
    "    y_hat = # Get the predicted value of y Hint: y_hat = x dot w (use np.dot)\n",
    "    # Cost function is given above but use 1/(2*n)sum((y_hat - y)^2)\n",
    "    cost = # find cost function Hint : np.sum will be useful\n",
    "    return cost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)   # do not change seed otherwise answer will not match \n",
    "x_temp = np.random.rand(5,5)\n",
    "y_temp = np.random.rand(5,1)\n",
    "\n",
    "w_temp = np.random.rand(6,1)  # using random weights for illustration purpose\n",
    "x_temp = normalize(x_temp)\n",
    "cost = compute_cost(x_temp,y_temp,w_temp)\n",
    "\n",
    "print(cost)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> The answer should match this </h2>\n",
    "\n",
    "0.2911040132454303"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> Gradient descent optimization algorithm </h1>\n",
    "\n",
    "Gradient Descent is the most common optimization algorithm in machine learning and deep learning. It is a first-order optimization algorithm. This means it only takes into account the first derivative when performing the updates on the parameters. On each iteration, we update the parameters in the opposite direction of the gradient of the objective function J(w) w.r.t the parameters where the gradient gives the direction of the steepest ascent. The size of the step we take on each iteration to reach the local minimum is determined by the learning rate α. Therefore, we follow the direction of the slope downhill until we reach a local minimum.\n",
    "\n",
    "<img src = \"assets/Images/Cost-Function.jpg\" />\n",
    "<br>\n",
    "<img src = \"assets/Images/gradiant_descent.jpg\" />\n",
    "\n",
    "<br>\n",
    "<h1> Illustration of gradient descent by valley descent example </h1>\n",
    "<br>\n",
    "<img src = \"assets/Images/gd_illu.jpeg\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def gradient_descent(x,y,w,iterations,learning_rate):\n",
    "    no_sample = #initialize no_sample as above mentioned \n",
    "    j = []\n",
    "    initial_cost = # use compute_cost\n",
    "    j.append(initial_cost)\n",
    "    \n",
    "    #The idea is to update w at every iteration by gradient descent algorithm\n",
    "    \n",
    "    for i in range(iterations):\n",
    "        y_hat = # calculate predicted values of y\n",
    "        \n",
    "        # Now update weight from equation w = w - (learning_rate/no_sample)*(transpose(x) dot (y_hat - y))\n",
    "        w = # use np.dot and x.T for transpose\n",
    "        cost = # use compute_cost\n",
    "        j.append(cost)\n",
    "    return (w,j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)   # do not change seed otherwise answer will not match \n",
    "x_temp = np.random.rand(5,5)\n",
    "y_temp = np.random.rand(5,1)\n",
    "\n",
    "w_temp = np.random.rand(6,1)  # using random weights for illustration purpose\n",
    "x_temp = normalize(x_temp)\n",
    "\n",
    "w_temp,j_temp = gradient_descent(x_temp,y_temp,w_temp,1000,0.001)\n",
    "print(\"Weights after updation: {}\".format(w_temp))\n",
    "print(\"Initial cost: {}\".format(j_temp[0]))\n",
    "print(\"final cost after updation: {}\".format(j_temp[-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> The answer should match this </h2>\n",
    "\n",
    "Weights after updation: [[ 0.43422511]\n",
    "\n",
    " [ 0.6546597 ]\n",
    " \n",
    " [ 0.31910619]\n",
    " \n",
    " [ 0.34074398]\n",
    " \n",
    " [-0.04576342]\n",
    " \n",
    " [ 0.6052419 ]]\n",
    " \n",
    "Initial cost: 0.2911040132454303\n",
    "\n",
    "final cost after updation: 0.1679955760288899"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3> Training time </h3>\n",
    "We are setting learning rate to 0.001 you can tweak the learning rate and check the results for your self, Learning rate is known as hyperparameter in machine learning tearms and setting up optimum learning rate is quite a challanging task some times.\n",
    "\n",
    "Generally we split data in train set and validation set by using sklearn.model_selection.train_test_split(x,y) but here we have very less training example so we are going to use all data for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iteration = 4000\n",
    "lr = 0.001\n",
    "x = normalize(x)\n",
    "w,j = gradient_descent(x,y,w,iteration,lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2> Cost convergence graph </h2>\n",
    "\n",
    "As we can see from the below graph, the cost is decreasing with each iteration, the graph below shows us that the training is stable and we have selected optimum learning rate, if cost in graph overshoots or increase with time in some iterations then we know that the learning rate is not optimum and we have to change it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = np.arange(iteration + 1)\n",
    "\n",
    "plt.figure(figsize = (16,12),dpi = 96)\n",
    "plt.plot(t,j)\n",
    "plt.xlabel('iterations')\n",
    "plt.ylabel('cost')\n",
    "plt.title('convergence graph of cost function')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Initial cost = {}'.format(j[0]))\n",
    "print('final cost = {}'.format(j[-1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can not simply measure accuracy by checking how much predictions are correct as we can in classification problem, so we have to device some method to measure accuracy. we are going to use equation 1 - (sum((y-y_hat)2)/sum((y-mean(y))2) for the accuracy measurement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score(x,y,w):\n",
    "    x = normalize(x)\n",
    "    y = np.expand_dims(y,axis = 1)\n",
    "    \n",
    "    y_hat = np.dot(x,w)\n",
    "    score = 1.0 - ((np.sum((y - y_hat)**2))/(np.sum((y - np.mean(y))**2)))\n",
    "    return score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test = dataset.data\n",
    "y_test = dataset.target\n",
    "\n",
    "s = score(x_test,y_test,w)\n",
    "print('score of the system is : {}'.format(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have got quite a good accuracy with such a simple model on this multi featured dataset. quite an accomplishment cheer up!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(x,w):\n",
    "    n_samples = x.shape[0]\n",
    "    x = normalize(x)\n",
    "    \n",
    "    y = np.dot(x,w)\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test = dataset.data\n",
    "y = predict(x_test,w)\n",
    "\n",
    "#print(y) uncomment the commented value to see values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(dataset.target)   uncomment the commented value to see values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So you can check predicted y values and also original target values they are very near to each other, so we have trained a good model with such a few lines of code! That's the power of machine learning and you are all set to implement this knowledge to tackle a real world problem!\n",
    "\n",
    "Cheers! see you next time!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}