Advanced Numerical Linear Algebra/week_6_health_outcomes_with_linear_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Health Outcomes with Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing basic lib\n",
    "from sklearn import datasets, linear_model, metrics\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "import math, scipy, numpy as np   \n",
    "from scipy import linalg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Diabetes Dataset\n",
    "        We will use a dataset from patients with diabetes. The data consists of 442 samples and 10 variables (like tall and skinny). The dependent variable is a quantitatve measure of disease progeression one year after the basel\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "        This is a clasical dataset, famously used by Efron, Hastie, Johnstone, and Tibshirani in their Least angle regression paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = datasets.load_diabetes()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_names = ['age','sex','bmi','bp','s1','s2','s3','s4','s5','s6']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn,test, y_trn, y_test = train_test_split(data.data, data.target, test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((353, 10), (89, 10))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn.shape, test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression in Sci-kit Learn\n",
    "\n",
    "Consider a system $X\\beta = y$ , where $X$ has more rows than columns. This occurs when you have more data samples than variables. We want to find $/beta$ that minimizes\n",
    "\n",
    "$$ || X\\beta - y ||_2 $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "        Starting with simmpler sklearn implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slowest run took 37.80 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "12.5 ms ± 20.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "regr = linear_model.LinearRegression()\n",
    "%timeit regr.fit(trn, y_trn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = regr.predict(test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "        We will have some metrics on how good our prediction is. We will look at the mean squared norm (L2) and mean absolute error (L1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def regr_metric(act, pred):\n",
    "    return (math.sqrt(metrics.mean_squared_error(act, pred)), metrics.mean_absolute_error(act,pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(49.876877547419426, 39.34822094080999)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr_metric(y_test, regr.predict(test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polynomial Features\n",
    "\n",
    "Linear Regression finds the best coefficient $/beta_i$ for:\n",
    "$$ x_0 \\beta_0 + x_1 \\beta_1 + x_2\\beta_2 =y $$\n",
    "\n",
    "Adding Polynomial features is still alinear regression problem, just with more terms:\n",
    "\n",
    "$$x_0 \\beta_0 + x_1 \\beta_1 + x_2 \\beta_2 + x_o^2 \\beta_3 + x_{0} x_1 \\beta_4 + .... = y $$\n",
    "\n",
    "    We need to use our original data X to calculate the additional polynomial features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(353, 10)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "            The perfomrmce of the model was bad, now to improve it, we have to add some features. Currenlty our model is linear in each variable, but we can add polynomial features to change this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "poly = PolynomialFeatures(include_bias= False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn_feat = poly.fit_transform(trn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'age, sex, bmi, bp, s1, s2, s3, s4, s5, s6, age^2, age sex, age bmi, age bp, age s1, age s2, age s3, age s4, age s5, age s6, sex^2, sex bmi, sex bp, sex s1, sex s2, sex s3, sex s4, sex s5, sex s6, bmi^2, bmi bp, bmi s1, bmi s2, bmi s3, bmi s4, bmi s5, bmi s6, bp^2, bp s1, bp s2, bp s3, bp s4, bp s5, bp s6, s1^2, s1 s2, s1 s3, s1 s4, s1 s5, s1 s6, s2^2, s2 s3, s2 s4, s2 s5, s2 s6, s3^2, s3 s4, s3 s5, s3 s6, s4^2, s4 s5, s4 s6, s5^2, s5 s6, s6^2'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "', '.join(poly.get_feature_names_out(feature_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(353, 65)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_feat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## now do the fitting\n",
    "\n",
    "regr.fit(trn_feat, y_trn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56.08366396156073, 42.37826260086632)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr_metric(y_test, regr.predict(poly.fit_transform(test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "        Since time is squared in features and linear in points, the below code will be slow to run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5.03 ms ± 1.02 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit poly.fit_transform(trn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Speeding up feature generation\n",
    "\n",
    "        We would like to speed this up. we will use Numba, a python library that  compiles code directly into C\n",
    "\n",
    "\n",
    "        Numba is a compiler\n",
    "\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "        Experiments with vectorization and native code\n",
    "\n",
    "        Let's get aquainted with numba       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math, numpy as np, matplotlib.pyplot as plt\n",
    "from pandas_summary import DataFrameSummary\n",
    "from scipy import ndimage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numba import jit, vectorize, guvectorize, cuda, float32, void, float64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "       We will show the impact of:\n",
    "       1. Avoid memory allocations and copies\n",
    "       2. Better locality\n",
    "       3. Vectorization\n",
    "\n",
    "    If we use numpy on whole arrays at a time, it creates lots of temporaries, and can't use cache. If we use numba looping through an array item at a time, then we don't have to allocate large temporary arrays, and can reuse cached data since we're doing multiple calculations on each array item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# untype and unvectroized\n",
    "def proc_python(xx,yy):\n",
    "    zz = np.zeros(nobs, dtype='float32')\n",
    "    for j in range(nobs):\n",
    "        x,y = xx[j], yy[j]\n",
    "        x = x*2 - (y *55)\n",
    "\n",
    "        y = x + y*2\n",
    "\n",
    "        z = x+y + 99\n",
    "        z = z* (z - .88)\n",
    "        zz[j] = z\n",
    "    return zz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobs = 10000\n",
    "x = np.random.randn(nobs).astype('float32')\n",
    "y = np.random.randn(nobs).astype('float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.35 s ± 574 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit proc_python(x,y)\n",
    "\n",
    "# this is the untyped and unvectorized version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy \n",
    "         We will vectorize it now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Typed and Vectorized\n",
    "def proc_numpy(x,y):\n",
    "    z = np.zeros(nobs, dtype='float32')\n",
    "    x = x*2 - ( y * 55 )\n",
    "    y = x + y*2         \n",
    "    z = x + y + 99      \n",
    "    z = z * ( z - .88 ) \n",
    "    return z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose( proc_numpy(x,y), proc_python(x,y), atol=1e-4 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "641 µs ± 125 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit proc_numpy(x,y)    # Typed and vectorized"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numba\n",
    "\n",
    "        Numab offers several different decorators. We will try two different ones:\n",
    "        1. @jit : very general\n",
    "        2. @vectorize : don't need to write a for loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "@jit()\n",
    "def proc_numba(xx,yy,zz):\n",
    "    for j in range(nobs):   \n",
    "        x, y = xx[j], yy[j] \n",
    "        x = x*2 - ( y * 55 )\n",
    "        y = x + y*2         \n",
    "        z = x + y + 99      \n",
    "        z = z * ( z - .88 ) \n",
    "        zz[j] = z           \n",
    "    return zz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = np.zeros(nobs).astype('float32')\n",
    "np.allclose( proc_numpy(x,y), proc_numba(x,y,z), atol=1e-4 )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "95.8 µs ± 21.6 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit proc_numba(x,y,z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "                Now we'll use the vectorize decorator. \n",
    "                Numba's compiler optimizes this in a smarter way\n",
    "\n",
    "\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "@vectorize\n",
    "def vec_numba(x,y):\n",
    "    x = x*2 - ( y * 55 )\n",
    "    y = x + y*2         \n",
    "    z = x + y + 99      \n",
    "    return z * ( z - .88 ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(vec_numba(x,y), proc_numba(x,y,z), atol=1e-4 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "104 µs ± 32.7 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit vec_numba(x,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "        NUMBA is amazingly fast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numba Polynomial features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "@jit(nopython=True)\n",
    "def vec_poly(x, res):\n",
    "    m,n=x.shape\n",
    "    feat_idx=0\n",
    "    for i in range(n):\n",
    "        v1=x[:,i]\n",
    "        for k in range(m): res[k,feat_idx] = v1[k]\n",
    "        feat_idx+=1\n",
    "        for j in range(i,n):\n",
    "            for k in range(m): res[k,feat_idx] = v1[k]*x[k,j]\n",
    "            feat_idx+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Row Major vs Column Major Storage\n",
    "\n",
    "        \"The row-major layout of a matrix puts the first row in contiguous memory, then the second row right after it, then the third, and so on. Column-major layout puts the first column in contiguous memory, then the second, etc.... While knowing which layout a particular data set is using is critical for good performance, there's no single answer to the question which layout 'is better' in general.\n",
    "\n",
    "        \"It turns out that matching the way your algorithm works with the data layout can make or break the performance of an application.\n",
    "\n",
    "        \"The short takeaway is: always traverse the data in the order it was laid out.\"\n",
    "\n",
    "        Column-major layout: Fortran, Matlab, R, and Julia\n",
    "\n",
    "        Row-major layout: C, C++, Python, Pascal, Mathematica\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "trn = np.asfortranarray(trn)\n",
    "test = np.asfortranarray(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "m,n = trn.shape\n",
    "n_feat = n*(n+1)//2 + n\n",
    "trn_feat = np.zeros((m,n_feat), order='F')\n",
    "\n",
    "test_feat = np.zeros((len(y_test), n_feat), order='F')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "vec_poly(trn,trn_feat)\n",
    "vec_poly(test,test_feat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr.fit(trn_feat, y_trn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56.08366396156081, 42.37826260086712)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr_metric(y_test, regr.predict(test_feat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "405 µs ± 125 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit vec_poly(trn, trn_feat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    This was the time from the scikit learn implementation PolynomialFeatures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The slowest run took 5.93 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "2.12 ms ± 1.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit poly.fit_transform(trn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regularization and Noise\n",
    "\n",
    "        Regularization is a way to reduce over-fitting and create models that bettter generalize to new dataLasso Regression uses an L1 penalty , which pushestoward sparse coefficients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg_regr = linear_model.LassoCV(n_alphas= 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\Monit Sharma\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\sklearn\\linear_model\\_coordinate_descent.py:644: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 93470.86108051939, tolerance: 178.35448262411347\n",
      "  positive,\n",
      "c:\\Users\\Monit Sharma\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\sklearn\\linear_model\\_coordinate_descent.py:644: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 43641.458391502674, tolerance: 177.24584452296824\n",
      "  positive,\n",
      "c:\\Users\\Monit Sharma\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\sklearn\\linear_model\\_coordinate_descent.py:644: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 12906.598891411093, tolerance: 181.50651166077742\n",
      "  positive,\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LassoCV(n_alphas=10)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg_regr.fit(trn_feat, y_trn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.011006106192913239"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg_regr.alpha_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(49.58213957493595, 39.277438762354656)"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr_metric(y_test, reg_regr.predict(test_feat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Noise \n",
    "\n",
    "    Adding some noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "idxs = np.random.randint(0, len(trn), 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_trn2 = np.copy(y_trn)\n",
    "y_trn2[idxs] *= 10 # label noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(49.87687754741944, 39.34822094080999)"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr = linear_model.LinearRegression()\n",
    "regr.fit(trn, y_trn)\n",
    "regr_metric(y_test, regr.predict(test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(69.14132406238272, 57.59514125171805)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "regr.fit(trn, y_trn2)\n",
    "regr_metric(y_test, regr.predict(test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "            Huber Loss is a loss function that is less sensitive to outliers than squared error loss. It is quadratic for small error values, and linear for large values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\Monit Sharma\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\sklearn\\linear_model\\_huber.py:332: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(51.24180446481409, 40.38448625427875)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hregr = linear_model.HuberRegressor()\n",
    "hregr.fit(trn, y_trn2)\n",
    "regr_metric(y_test, hregr.predict(test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.0 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "c0f2ef85a013b3e049ebe5ad4c39b42bd34e85c4b0686b7bef6ce5ff62aa91e1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}