Predicting House Prices Using Linear Regression .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting House Prices Using Multiple Linear Regression "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project uses multiple linear regression to predict house prices in the city of Ames, Iowa, United States. The data with which this project is based on is available here: https://www.kaggle.com/marcopale/housing?select=AmesHousing.csv\n",
    "\n",
    "The model is also tested via k-fold cross validation where we find that the optimal number of k-folds for our regression model is 4. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.options.display.max_columns = 999\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import KFold\n",
    "import seaborn as sns\n",
    "\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn import linear_model\n",
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"AmesHousing.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Order</th>\n",
       "      <th>PID</th>\n",
       "      <th>MS SubClass</th>\n",
       "      <th>MS Zoning</th>\n",
       "      <th>Lot Frontage</th>\n",
       "      <th>Lot Area</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>Lot Shape</th>\n",
       "      <th>Land Contour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>Lot Config</th>\n",
       "      <th>Land Slope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Condition 1</th>\n",
       "      <th>Condition 2</th>\n",
       "      <th>Bldg Type</th>\n",
       "      <th>House Style</th>\n",
       "      <th>Overall Qual</th>\n",
       "      <th>Overall Cond</th>\n",
       "      <th>Year Built</th>\n",
       "      <th>Year Remod/Add</th>\n",
       "      <th>Roof Style</th>\n",
       "      <th>Roof Matl</th>\n",
       "      <th>Exterior 1st</th>\n",
       "      <th>Exterior 2nd</th>\n",
       "      <th>Mas Vnr Type</th>\n",
       "      <th>Mas Vnr Area</th>\n",
       "      <th>Exter Qual</th>\n",
       "      <th>Exter Cond</th>\n",
       "      <th>Foundation</th>\n",
       "      <th>Bsmt Qual</th>\n",
       "      <th>Bsmt Cond</th>\n",
       "      <th>Bsmt Exposure</th>\n",
       "      <th>BsmtFin Type 1</th>\n",
       "      <th>BsmtFin SF 1</th>\n",
       "      <th>BsmtFin Type 2</th>\n",
       "      <th>BsmtFin SF 2</th>\n",
       "      <th>Bsmt Unf SF</th>\n",
       "      <th>Total Bsmt SF</th>\n",
       "      <th>Heating</th>\n",
       "      <th>Heating QC</th>\n",
       "      <th>Central Air</th>\n",
       "      <th>Electrical</th>\n",
       "      <th>1st Flr SF</th>\n",
       "      <th>2nd Flr SF</th>\n",
       "      <th>Low Qual Fin SF</th>\n",
       "      <th>Gr Liv Area</th>\n",
       "      <th>Bsmt Full Bath</th>\n",
       "      <th>Bsmt Half Bath</th>\n",
       "      <th>Full Bath</th>\n",
       "      <th>Half Bath</th>\n",
       "      <th>Bedroom AbvGr</th>\n",
       "      <th>Kitchen AbvGr</th>\n",
       "      <th>Kitchen Qual</th>\n",
       "      <th>TotRms AbvGrd</th>\n",
       "      <th>Functional</th>\n",
       "      <th>Fireplaces</th>\n",
       "      <th>Fireplace Qu</th>\n",
       "      <th>Garage Type</th>\n",
       "      <th>Garage Yr Blt</th>\n",
       "      <th>Garage Finish</th>\n",
       "      <th>Garage Cars</th>\n",
       "      <th>Garage Area</th>\n",
       "      <th>Garage Qual</th>\n",
       "      <th>Garage Cond</th>\n",
       "      <th>Paved Drive</th>\n",
       "      <th>Wood Deck SF</th>\n",
       "      <th>Open Porch SF</th>\n",
       "      <th>Enclosed Porch</th>\n",
       "      <th>3Ssn Porch</th>\n",
       "      <th>Screen Porch</th>\n",
       "      <th>Pool Area</th>\n",
       "      <th>Pool QC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>Misc Feature</th>\n",
       "      <th>Misc Val</th>\n",
       "      <th>Mo Sold</th>\n",
       "      <th>Yr Sold</th>\n",
       "      <th>Sale Type</th>\n",
       "      <th>Sale Condition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>526301100</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>141.0</td>\n",
       "      <td>31770</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NAmes</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>1Story</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>1960</td>\n",
       "      <td>1960</td>\n",
       "      <td>Hip</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>Plywood</td>\n",
       "      <td>Stone</td>\n",
       "      <td>112.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>CBlock</td>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "      <td>BLQ</td>\n",
       "      <td>639.0</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0.0</td>\n",
       "      <td>441.0</td>\n",
       "      <td>1080.0</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Fa</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1656</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1656</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>7</td>\n",
       "      <td>Typ</td>\n",
       "      <td>2</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>1960.0</td>\n",
       "      <td>Fin</td>\n",
       "      <td>2.0</td>\n",
       "      <td>528.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>P</td>\n",
       "      <td>210</td>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>215000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>526350040</td>\n",
       "      <td>20</td>\n",
       "      <td>RH</td>\n",
       "      <td>80.0</td>\n",
       "      <td>11622</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NAmes</td>\n",
       "      <td>Feedr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>1Story</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>1961</td>\n",
       "      <td>1961</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>CBlock</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>No</td>\n",
       "      <td>Rec</td>\n",
       "      <td>468.0</td>\n",
       "      <td>LwQ</td>\n",
       "      <td>144.0</td>\n",
       "      <td>270.0</td>\n",
       "      <td>882.0</td>\n",
       "      <td>GasA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>896</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>896</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>5</td>\n",
       "      <td>Typ</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>1961.0</td>\n",
       "      <td>Unf</td>\n",
       "      <td>1.0</td>\n",
       "      <td>730.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>140</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>120</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MnPrv</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>105000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>526351010</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>81.0</td>\n",
       "      <td>14267</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NAmes</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>1Story</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>1958</td>\n",
       "      <td>1958</td>\n",
       "      <td>Hip</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>Wd Sdng</td>\n",
       "      <td>Wd Sdng</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>108.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>CBlock</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>No</td>\n",
       "      <td>ALQ</td>\n",
       "      <td>923.0</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0.0</td>\n",
       "      <td>406.0</td>\n",
       "      <td>1329.0</td>\n",
       "      <td>GasA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>1329</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1329</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>1958.0</td>\n",
       "      <td>Unf</td>\n",
       "      <td>1.0</td>\n",
       "      <td>312.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>393</td>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Gar2</td>\n",
       "      <td>12500</td>\n",
       "      <td>6</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>172000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>526353030</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>93.0</td>\n",
       "      <td>11160</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NAmes</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>1Story</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1968</td>\n",
       "      <td>1968</td>\n",
       "      <td>Hip</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>BrkFace</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>CBlock</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>No</td>\n",
       "      <td>ALQ</td>\n",
       "      <td>1065.0</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1045.0</td>\n",
       "      <td>2110.0</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Ex</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>2110</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2110</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>Ex</td>\n",
       "      <td>8</td>\n",
       "      <td>Typ</td>\n",
       "      <td>2</td>\n",
       "      <td>TA</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>1968.0</td>\n",
       "      <td>Fin</td>\n",
       "      <td>2.0</td>\n",
       "      <td>522.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>244000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>527105010</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>74.0</td>\n",
       "      <td>13830</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Gilbert</td>\n",
       "      <td>Norm</td>\n",
       "      <td>Norm</td>\n",
       "      <td>1Fam</td>\n",
       "      <td>2Story</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>1997</td>\n",
       "      <td>1998</td>\n",
       "      <td>Gable</td>\n",
       "      <td>CompShg</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>VinylSd</td>\n",
       "      <td>None</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>PConc</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>No</td>\n",
       "      <td>GLQ</td>\n",
       "      <td>791.0</td>\n",
       "      <td>Unf</td>\n",
       "      <td>0.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>928.0</td>\n",
       "      <td>GasA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Y</td>\n",
       "      <td>SBrkr</td>\n",
       "      <td>928</td>\n",
       "      <td>701</td>\n",
       "      <td>0</td>\n",
       "      <td>1629</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>6</td>\n",
       "      <td>Typ</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>1997.0</td>\n",
       "      <td>Fin</td>\n",
       "      <td>2.0</td>\n",
       "      <td>482.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>212</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>MnPrv</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>2010</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>189900</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Order        PID  MS SubClass MS Zoning  Lot Frontage  Lot Area Street  \\\n",
       "0      1  526301100           20        RL         141.0     31770   Pave   \n",
       "1      2  526350040           20        RH          80.0     11622   Pave   \n",
       "2      3  526351010           20        RL          81.0     14267   Pave   \n",
       "3      4  526353030           20        RL          93.0     11160   Pave   \n",
       "4      5  527105010           60        RL          74.0     13830   Pave   \n",
       "\n",
       "  Alley Lot Shape Land Contour Utilities Lot Config Land Slope Neighborhood  \\\n",
       "0   NaN       IR1          Lvl    AllPub     Corner        Gtl        NAmes   \n",
       "1   NaN       Reg          Lvl    AllPub     Inside        Gtl        NAmes   \n",
       "2   NaN       IR1          Lvl    AllPub     Corner        Gtl        NAmes   \n",
       "3   NaN       Reg          Lvl    AllPub     Corner        Gtl        NAmes   \n",
       "4   NaN       IR1          Lvl    AllPub     Inside        Gtl      Gilbert   \n",
       "\n",
       "  Condition 1 Condition 2 Bldg Type House Style  Overall Qual  Overall Cond  \\\n",
       "0        Norm        Norm      1Fam      1Story             6             5   \n",
       "1       Feedr        Norm      1Fam      1Story             5             6   \n",
       "2        Norm        Norm      1Fam      1Story             6             6   \n",
       "3        Norm        Norm      1Fam      1Story             7             5   \n",
       "4        Norm        Norm      1Fam      2Story             5             5   \n",
       "\n",
       "   Year Built  Year Remod/Add Roof Style Roof Matl Exterior 1st Exterior 2nd  \\\n",
       "0        1960            1960        Hip   CompShg      BrkFace      Plywood   \n",
       "1        1961            1961      Gable   CompShg      VinylSd      VinylSd   \n",
       "2        1958            1958        Hip   CompShg      Wd Sdng      Wd Sdng   \n",
       "3        1968            1968        Hip   CompShg      BrkFace      BrkFace   \n",
       "4        1997            1998      Gable   CompShg      VinylSd      VinylSd   \n",
       "\n",
       "  Mas Vnr Type  Mas Vnr Area Exter Qual Exter Cond Foundation Bsmt Qual  \\\n",
       "0        Stone         112.0         TA         TA     CBlock        TA   \n",
       "1         None           0.0         TA         TA     CBlock        TA   \n",
       "2      BrkFace         108.0         TA         TA     CBlock        TA   \n",
       "3         None           0.0         Gd         TA     CBlock        TA   \n",
       "4         None           0.0         TA         TA      PConc        Gd   \n",
       "\n",
       "  Bsmt Cond Bsmt Exposure BsmtFin Type 1  BsmtFin SF 1 BsmtFin Type 2  \\\n",
       "0        Gd            Gd            BLQ         639.0            Unf   \n",
       "1        TA            No            Rec         468.0            LwQ   \n",
       "2        TA            No            ALQ         923.0            Unf   \n",
       "3        TA            No            ALQ        1065.0            Unf   \n",
       "4        TA            No            GLQ         791.0            Unf   \n",
       "\n",
       "   BsmtFin SF 2  Bsmt Unf SF  Total Bsmt SF Heating Heating QC Central Air  \\\n",
       "0           0.0        441.0         1080.0    GasA         Fa           Y   \n",
       "1         144.0        270.0          882.0    GasA         TA           Y   \n",
       "2           0.0        406.0         1329.0    GasA         TA           Y   \n",
       "3           0.0       1045.0         2110.0    GasA         Ex           Y   \n",
       "4           0.0        137.0          928.0    GasA         Gd           Y   \n",
       "\n",
       "  Electrical  1st Flr SF  2nd Flr SF  Low Qual Fin SF  Gr Liv Area  \\\n",
       "0      SBrkr        1656           0                0         1656   \n",
       "1      SBrkr         896           0                0          896   \n",
       "2      SBrkr        1329           0                0         1329   \n",
       "3      SBrkr        2110           0                0         2110   \n",
       "4      SBrkr         928         701                0         1629   \n",
       "\n",
       "   Bsmt Full Bath  Bsmt Half Bath  Full Bath  Half Bath  Bedroom AbvGr  \\\n",
       "0             1.0             0.0          1          0              3   \n",
       "1             0.0             0.0          1          0              2   \n",
       "2             0.0             0.0          1          1              3   \n",
       "3             1.0             0.0          2          1              3   \n",
       "4             0.0             0.0          2          1              3   \n",
       "\n",
       "   Kitchen AbvGr Kitchen Qual  TotRms AbvGrd Functional  Fireplaces  \\\n",
       "0              1           TA              7        Typ           2   \n",
       "1              1           TA              5        Typ           0   \n",
       "2              1           Gd              6        Typ           0   \n",
       "3              1           Ex              8        Typ           2   \n",
       "4              1           TA              6        Typ           1   \n",
       "\n",
       "  Fireplace Qu Garage Type  Garage Yr Blt Garage Finish  Garage Cars  \\\n",
       "0           Gd      Attchd         1960.0           Fin          2.0   \n",
       "1          NaN      Attchd         1961.0           Unf          1.0   \n",
       "2          NaN      Attchd         1958.0           Unf          1.0   \n",
       "3           TA      Attchd         1968.0           Fin          2.0   \n",
       "4           TA      Attchd         1997.0           Fin          2.0   \n",
       "\n",
       "   Garage Area Garage Qual Garage Cond Paved Drive  Wood Deck SF  \\\n",
       "0        528.0          TA          TA           P           210   \n",
       "1        730.0          TA          TA           Y           140   \n",
       "2        312.0          TA          TA           Y           393   \n",
       "3        522.0          TA          TA           Y             0   \n",
       "4        482.0          TA          TA           Y           212   \n",
       "\n",
       "   Open Porch SF  Enclosed Porch  3Ssn Porch  Screen Porch  Pool Area Pool QC  \\\n",
       "0             62               0           0             0          0     NaN   \n",
       "1              0               0           0           120          0     NaN   \n",
       "2             36               0           0             0          0     NaN   \n",
       "3              0               0           0             0          0     NaN   \n",
       "4             34               0           0             0          0     NaN   \n",
       "\n",
       "   Fence Misc Feature  Misc Val  Mo Sold  Yr Sold Sale Type Sale Condition  \\\n",
       "0    NaN          NaN         0        5     2010       WD          Normal   \n",
       "1  MnPrv          NaN         0        6     2010       WD          Normal   \n",
       "2    NaN         Gar2     12500        6     2010       WD          Normal   \n",
       "3    NaN          NaN         0        4     2010       WD          Normal   \n",
       "4  MnPrv          NaN         0        3     2010       WD          Normal   \n",
       "\n",
       "   SalePrice  \n",
       "0     215000  \n",
       "1     105000  \n",
       "2     172000  \n",
       "3     244000  \n",
       "4     189900  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin by defining some functions which will allow us to quickly iterate on different models. The functions will allow us to return a dataframe, select the features we are interested in and then train and test a linear regression model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "57088.25161263909"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Transform features function returns the train data frame. \n",
    "def transform_features(df):\n",
    "    return df\n",
    "\n",
    "# Select features function returns the \"Gr Liv Area\" and \"SalePrice\" columns from the train data frame.  \n",
    "def select_features(df):\n",
    "    return df[[\"Gr Liv Area\", \"SalePrice\"]]\n",
    "\n",
    "# Train and test function selects the first 1460 rows from df and assigns it to train with the remaining rows assigned to test. \n",
    "# The function trains a model using all numerical columns except the target column ('SalePrice') from the df returned from select_features()\n",
    "# The function then tests the model on the test set and returns the root mean squared error (RMSE) value. \n",
    "\n",
    "def train_and_test(df):  \n",
    "    train = df[:1460]\n",
    "    test = df[1460:]\n",
    "    \n",
    "    ## You can use `pd.DataFrame.select_dtypes()` to specify column types\n",
    "    ## and return only those columns as a data frame.\n",
    "    numeric_train = train.select_dtypes(include=['integer', 'float'])\n",
    "    numeric_test = test.select_dtypes(include=['integer', 'float'])\n",
    "    \n",
    "    ## You can use `pd.Series.drop()` to drop a value.\n",
    "    features = numeric_train.columns.drop(\"SalePrice\")\n",
    "    lr = linear_model.LinearRegression()\n",
    "    lr.fit(train[features], train[\"SalePrice\"])\n",
    "    predictions = lr.predict(test[features])\n",
    "    mse = mean_squared_error(test[\"SalePrice\"], predictions)\n",
    "    rmse = np.sqrt(mse)\n",
    "    \n",
    "    return rmse\n",
    "\n",
    "transform_df = transform_features(df)\n",
    "filtered_df = select_features(transform_df)\n",
    "rmse = train_and_test(filtered_df)\n",
    "\n",
    "rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning and Feature Engineering "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now remove any features whose number of missing values is beyond our cutoff point, dive deeper into potential categorical features anf transform text and numerical columns. \n",
    "\n",
    "Handle missing values:\n",
    "- All columns:\n",
    "  - Drop any with 5% or more missing values.\n",
    "- Text columns:\n",
    "  - Drop any with 1 or more missing values.\n",
    "- Numerical columns:\n",
    "  - For columns with missing values, fill in with the most common value in that column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1: All columns: Drop any with 5% or more missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Series object: column name -> number of missing values\n",
    "num_missing = df.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Filter Series to columns containing >5% missing values\n",
    "drop_missing_cols = num_missing[(num_missing > len(df)/20)].sort_values()\n",
    "\n",
    "# Drop those columns from the data frame. Note the use of the .index accessor\n",
    "df = df.drop(drop_missing_cols.index, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2: Text columns: Drop any with 1 or more missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Series object: column name -> number of missing values\n",
    "text_mv_counts = df.select_dtypes(include=['object']).isnull().sum().sort_values(ascending=False)\n",
    "\n",
    "# Filter Series to columns containing *any* missing values\n",
    "drop_missing_cols_2 = text_mv_counts[text_mv_counts > 0]\n",
    "\n",
    "df = df.drop(drop_missing_cols_2.index, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3: Numerical columns: For columns with missing values, fill in with the most common value in that column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtFin SF 1       1\n",
       "BsmtFin SF 2       1\n",
       "Bsmt Unf SF        1\n",
       "Total Bsmt SF      1\n",
       "Garage Cars        1\n",
       "Garage Area        1\n",
       "Bsmt Full Bath     2\n",
       "Bsmt Half Bath     2\n",
       "Mas Vnr Area      23\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute column-wise missing value counts\n",
    "num_missing = df.select_dtypes(include=['int', 'float']).isnull().sum()\n",
    "fixable_numeric_cols = num_missing[(num_missing < len(df)/20) & (num_missing > 0)].sort_values()\n",
    "fixable_numeric_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'BsmtFin SF 1': 0.0,\n",
       " 'BsmtFin SF 2': 0.0,\n",
       " 'Bsmt Unf SF': 0.0,\n",
       " 'Total Bsmt SF': 0.0,\n",
       " 'Garage Cars': 2.0,\n",
       " 'Garage Area': 0.0,\n",
       " 'Bsmt Full Bath': 0.0,\n",
       " 'Bsmt Half Bath': 0.0,\n",
       " 'Mas Vnr Area': 0.0}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Compute the most common value for each column in `fixable_nmeric_missing_cols`.\n",
    "replacement_values_dict = df[fixable_numeric_cols.index].mode().to_dict(orient='records')[0]\n",
    "replacement_values_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use `pd.DataFrame.fillna()` to replace missing values.\n",
    "df = df.fillna(replacement_values_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    64\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verify that every column has 0 missing values\n",
    "df.isnull().sum().value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create new features to better capture the information in some of the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2180   -1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "years_sold = df['Yr Sold'] - df['Year Built']\n",
    "years_sold[years_sold < 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1702   -1\n",
       "2180   -2\n",
       "2181   -1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "years_since_remod = df['Yr Sold'] - df['Year Remod/Add']\n",
    "years_since_remod[years_since_remod < 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new columns\n",
    "df['Years Before Sale'] = years_sold\n",
    "df['Years Since Remod'] = years_since_remod\n",
    "\n",
    "# Drop rows with negative values for both of these new features\n",
    "df = df.drop([1702, 2180, 2181], axis=0)\n",
    "\n",
    "# No longer need original year columns\n",
    "df = df.drop([\"Year Built\", \"Year Remod/Add\"], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now drop columns that are not useful for ML or leak data about the final sale.\n",
    "\n",
    "PID is a district number and order is the order in which house sales have been listed. Both of these figures are irrelvant for predicting house price and are thus dropped. Month sold, year sold, sale condition and sale type leak info about the date and type of sale and are thus also removed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop columns that aren't useful for ML\n",
    "df = df.drop([\"PID\", \"Order\"], axis=1)\n",
    "\n",
    "# Drop columns that leak info about the final sale\n",
    "df = df.drop([\"Mo Sold\", \"Sale Condition\", \"Sale Type\", \"Yr Sold\"], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We incorporate all this data cleaning into the transform_features() function to allow for quick iteration through models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"AmesHousing.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55275.367312413066"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def transform_features(df):\n",
    "    num_missing = df.isnull().sum()\n",
    "    drop_missing_cols = num_missing[(num_missing > len(df)/20)].sort_values()\n",
    "    df = df.drop(drop_missing_cols.index, axis=1)\n",
    "    \n",
    "    text_mv_counts = df.select_dtypes(include=['object']).isnull().sum().sort_values(ascending=False)\n",
    "    drop_missing_cols_2 = text_mv_counts[text_mv_counts > 0]\n",
    "    df = df.drop(drop_missing_cols_2.index, axis=1)\n",
    "    \n",
    "    num_missing = df.select_dtypes(include=['int', 'float']).isnull().sum()\n",
    "    fixable_numeric_cols = num_missing[(num_missing < len(df)/20) & (num_missing > 0)].sort_values()\n",
    "    replacement_values_dict = df[fixable_numeric_cols.index].mode().to_dict(orient='records')[0]\n",
    "    df = df.fillna(replacement_values_dict)\n",
    "    \n",
    "    years_sold = df['Yr Sold'] - df['Year Built']\n",
    "    years_since_remod = df['Yr Sold'] - df['Year Remod/Add']\n",
    "    df['Years Before Sale'] = years_sold\n",
    "    df['Years Since Remod'] = years_since_remod\n",
    "    df = df.drop([1702, 2180, 2181], axis=0)\n",
    "\n",
    "    df = df.drop([\"PID\", \"Order\", \"Mo Sold\", \"Sale Condition\", \"Sale Type\", \"Year Built\", \"Year Remod/Add\"], axis=1)\n",
    "    return df\n",
    "\n",
    "\n",
    "df = pd.read_csv(\"AmesHousing.csv\")\n",
    "transform_df = transform_features(df)\n",
    "filtered_df = select_features(transform_df)\n",
    "rmse = train_and_test(filtered_df)\n",
    "\n",
    "rmse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have cleaned and transformed a lot of the features, we need to select numerical features for our regression model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Mas Vnr Area</th>\n",
       "      <th>BsmtFin SF 1</th>\n",
       "      <th>BsmtFin SF 2</th>\n",
       "      <th>Bsmt Unf SF</th>\n",
       "      <th>Total Bsmt SF</th>\n",
       "      <th>Bsmt Full Bath</th>\n",
       "      <th>Bsmt Half Bath</th>\n",
       "      <th>Garage Cars</th>\n",
       "      <th>Garage Area</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>112.0</td>\n",
       "      <td>639.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>441.0</td>\n",
       "      <td>1080.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>528.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>468.0</td>\n",
       "      <td>144.0</td>\n",
       "      <td>270.0</td>\n",
       "      <td>882.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>730.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>108.0</td>\n",
       "      <td>923.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>406.0</td>\n",
       "      <td>1329.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>312.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1065.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1045.0</td>\n",
       "      <td>2110.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>522.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>791.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>928.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>482.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Mas Vnr Area  BsmtFin SF 1  BsmtFin SF 2  Bsmt Unf SF  Total Bsmt SF  \\\n",
       "0         112.0         639.0           0.0        441.0         1080.0   \n",
       "1           0.0         468.0         144.0        270.0          882.0   \n",
       "2         108.0         923.0           0.0        406.0         1329.0   \n",
       "3           0.0        1065.0           0.0       1045.0         2110.0   \n",
       "4           0.0         791.0           0.0        137.0          928.0   \n",
       "\n",
       "   Bsmt Full Bath  Bsmt Half Bath  Garage Cars  Garage Area  \n",
       "0             1.0             0.0          2.0        528.0  \n",
       "1             0.0             0.0          1.0        730.0  \n",
       "2             0.0             0.0          1.0        312.0  \n",
       "3             1.0             0.0          2.0        522.0  \n",
       "4             0.0             0.0          2.0        482.0  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We select only numerical data columns. \n",
    "\n",
    "numerical_df = transform_df.select_dtypes(include=['int', 'float'])\n",
    "numerical_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SalePrice            1.000000\n",
       "Overall Qual         0.801206\n",
       "Gr Liv Area          0.717596\n",
       "Garage Cars          0.648361\n",
       "Total Bsmt SF        0.644012\n",
       "Garage Area          0.641425\n",
       "1st Flr SF           0.635185\n",
       "Years Before Sale    0.558979\n",
       "Full Bath            0.546118\n",
       "Years Since Remod    0.534985\n",
       "Mas Vnr Area         0.506983\n",
       "TotRms AbvGrd        0.498574\n",
       "Fireplaces           0.474831\n",
       "BsmtFin SF 1         0.439284\n",
       "Wood Deck SF         0.328183\n",
       "Open Porch SF        0.316262\n",
       "Name: SalePrice, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corr_coefficients = abs(transform_df.corr()['SalePrice']).sort_values(ascending = False)\n",
    "corr_coefficients[corr_coefficients>0.3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['SalePrice', 'Overall Qual', 'Gr Liv Area', 'Garage Cars',\n",
       "       'Total Bsmt SF', 'Garage Area', '1st Flr SF', 'Years Before Sale',\n",
       "       'Full Bath', 'Years Since Remod', 'Mas Vnr Area', 'TotRms AbvGrd',\n",
       "       'Fireplaces', 'BsmtFin SF 1', 'Wood Deck SF', 'Open Porch SF'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We keep columns with a correlation coefficient larger than 0.3 (an arbitrary figure, can be changed later).\n",
    "\n",
    "correlated_cols = corr_coefficients[corr_coefficients>0.3].index\n",
    "correlated_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SalePrice</th>\n",
       "      <th>Overall Qual</th>\n",
       "      <th>Gr Liv Area</th>\n",
       "      <th>Garage Cars</th>\n",
       "      <th>Total Bsmt SF</th>\n",
       "      <th>Garage Area</th>\n",
       "      <th>1st Flr SF</th>\n",
       "      <th>Years Before Sale</th>\n",
       "      <th>Full Bath</th>\n",
       "      <th>Years Since Remod</th>\n",
       "      <th>Mas Vnr Area</th>\n",
       "      <th>TotRms AbvGrd</th>\n",
       "      <th>Fireplaces</th>\n",
       "      <th>BsmtFin SF 1</th>\n",
       "      <th>Wood Deck SF</th>\n",
       "      <th>Open Porch SF</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>215000</td>\n",
       "      <td>6</td>\n",
       "      <td>1656</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1080.0</td>\n",
       "      <td>528.0</td>\n",
       "      <td>1656</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>112.0</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>639.0</td>\n",
       "      <td>210</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>105000</td>\n",
       "      <td>5</td>\n",
       "      <td>896</td>\n",
       "      <td>1.0</td>\n",
       "      <td>882.0</td>\n",
       "      <td>730.0</td>\n",
       "      <td>896</td>\n",
       "      <td>49</td>\n",
       "      <td>1</td>\n",
       "      <td>49</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>468.0</td>\n",
       "      <td>140</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>172000</td>\n",
       "      <td>6</td>\n",
       "      <td>1329</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1329.0</td>\n",
       "      <td>312.0</td>\n",
       "      <td>1329</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>52</td>\n",
       "      <td>108.0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>923.0</td>\n",
       "      <td>393</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244000</td>\n",
       "      <td>7</td>\n",
       "      <td>2110</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2110.0</td>\n",
       "      <td>522.0</td>\n",
       "      <td>2110</td>\n",
       "      <td>42</td>\n",
       "      <td>2</td>\n",
       "      <td>42</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>1065.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>189900</td>\n",
       "      <td>5</td>\n",
       "      <td>1629</td>\n",
       "      <td>2.0</td>\n",
       "      <td>928.0</td>\n",
       "      <td>482.0</td>\n",
       "      <td>928</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>791.0</td>\n",
       "      <td>212</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   SalePrice  Overall Qual  Gr Liv Area  Garage Cars  Total Bsmt SF  \\\n",
       "0     215000             6         1656          2.0         1080.0   \n",
       "1     105000             5          896          1.0          882.0   \n",
       "2     172000             6         1329          1.0         1329.0   \n",
       "3     244000             7         2110          2.0         2110.0   \n",
       "4     189900             5         1629          2.0          928.0   \n",
       "\n",
       "   Garage Area  1st Flr SF  Years Before Sale  Full Bath  Years Since Remod  \\\n",
       "0        528.0        1656                 50          1                 50   \n",
       "1        730.0         896                 49          1                 49   \n",
       "2        312.0        1329                 52          1                 52   \n",
       "3        522.0        2110                 42          2                 42   \n",
       "4        482.0         928                 13          2                 12   \n",
       "\n",
       "   Mas Vnr Area  TotRms AbvGrd  Fireplaces  BsmtFin SF 1  Wood Deck SF  \\\n",
       "0         112.0              7           2         639.0           210   \n",
       "1           0.0              5           0         468.0           140   \n",
       "2         108.0              6           0         923.0           393   \n",
       "3           0.0              8           2        1065.0             0   \n",
       "4           0.0              6           1         791.0           212   \n",
       "\n",
       "   Open Porch SF  \n",
       "0             62  \n",
       "1              0  \n",
       "2             36  \n",
       "3              0  \n",
       "4             34  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Drop columns with less than 0.4 correlation with SalePrice\n",
    "\n",
    "transform_df[correlated_cols].head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1ba44268a00>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAboAAAFQCAYAAADJKLthAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd5xV1bXHvz+KDIiisRcUNYhRVBTsDWN5JrE+NWhMImpCNJaYl2ZiYojGGBPfM9YYNIomFmIJmtgbVhRQOooasIAmiNiQOjPr/bH3hTN3buXcYQ6X9fVzPnPOPnuvvc9lvGvWPnv/lswMx3Ecx6lXOrT3ABzHcRynLXFH5ziO49Q17ugcx3GcusYdneM4jlPXuKNzHMdx6hp3dI7jOE5d447OcRzHWSlIulHSHElTityXpCslvSFpkqRda9GvOzrHcRxnZTEcOKzE/S8BveMxBPhjLTp1R+c4juOsFMzsaWBeiSpHAbdY4AVgHUmbpO23U1oDzspl6dwZqaVsGic+ls7AnNlph4DN+yC1DZqb09vIwBjWO+/B1DbeP3dAqvabXTUh9RhmDe6T2gbN6ZWabHFjuiEsSNceYP4MpbbR3JTexhbjHk9tpJrvnDU22OY7hEgsxzAzG1ZFd5sB7ySuZ8Wy96qw0Qp3dI7jOE5NiE6tGseWTyHHnPqvH3d0juM4TnGalq7M3mYBPRPXmwPvpjXq7+gcx3Gc4jQ3V36k5z7gm3H15Z7Ax2aWatoSVhNHJ+l8SVPjctUJkvYoUXe4pOPK2BsuaWa09bKkvYrUu1DSwWnH7ziO016YNVd8lEPS7cBooI+kWZJOk3S6pNNjlQeAGcAbwPXAd2vxDHU/dRmd0OHArma2WNL6wBo1MP0jM7tL0qHAn4Cd8vrtaGYX1KAfx3Gc9qOGi77M7MQy9w04s2YdRlaHiG4TYK6ZLQYws7lm9q6kCySNlTRF0jBJrV6CSuov6SlJL0l6uMgy16eBz8f6b0a7zwLHJ6NDSbtJel7SREljJK0lqaOk38dxTJL0nbb7GBzHcVYAa678yCirg6N7BOgp6TVJ10o6IJZfbWa7mVlfoCsh6luGpM7AVcBxZtYfuBG4uID9I4DJietFZravmd2RsLUGMAL4npntDBwMLAROI8xB7wbsBnxb0lb5HUgaImmcpHE33HL7Cn0IjuM4K0RzU+VHRqn7qUszmy+pP7AfcCAwQtJ5wKeSfgx0Az4HTAX+kWjaB+gLPBqDvY603Mvxe0k/B94nOKwcIwoMow/wnpmNjWP6BCBOe+6UeCfYg6AIMDPvGZYt2a3FPjrHcZyKaUq/r7C9qXtHB2BmTcAoYJSkycB3CO/UBpjZO5KGAg15zQRMNbOCC02I7+gKlH9WoEwU3gsi4Gwze7j8UziO46x8KllkknXqfupSUh9JvRNF/YDp8XyupO5AoVWW04ENcisqJXWWtMMKDuNVYFNJu0Vba0nqBDwMnBGnSZG0raQ1V7APx3Gc2rNytxe0CatDRNcduErSOkAjYdnqEOAjwru1N4Gx+Y3MbEmcUrxSUg/CZ/UHwhRnVURbg+I4uhLezx0M3AD0Al6Oi2HeB46u1r7jOE6bUQcRXd07OjN7Cdi7wK2fxyO//uDE+QRg/1J18sp7lbA1FtizQLOfxcNxHCd7ZHiRSaXUvaOrN1ILMgOddk63h33JlTXwy127pDZhn8xPbUOdOqZq3/Tuh6nHsLQWL/sXp5Npmj1kB9a/Np2w89LZhV5PV0fjJ6lN0CHlLtmP3sp/XV89S5em+70C6NKQkUUgvhjFcZx6IK2Tc+oYn7p0HMdx6poMLzKpFHd0juM4TlHC7qxVG3d0juM4TnHqYOoy8/voJG0u6V5Jr0v6l6QroqRWW/c7P/7sJWlKkTo7SHoiyov9S9KvJK3wZxq1Mtdf0faO4zg1pw720WXa0cW9ZfcAI82sN7AtYV9cIc3Jam2nimbjfrj7gN+a2bbAjsDuwPfSjs1xHCczNC2t/MgomXZ0wBcJIsk3wTIpr+8Dp0rqJunFpFqJpFEx48Cakm6MWQHGSzoq3h8s6U5J/wAekdRd0uMxp9zkXL0K+RrwnJk9Ese2ADgL+FHsa6ikHybGNkVSr3g+MmZEmCppyIp/PI7jOG2MZy9oc3YAXkoWREHktwmpce4AvgoQU+hsGjeInw88EbMCHEgQYM5Ja+0FnGxmXwQWAceY2a6x3v8WStdTxdj+BXSNKiylODVmRBgAnCNpvVKVk9kL/vzPpyscnuM4Tg2og6nLrC9GKSWGbMDfgEeBXxIc3p3x/qHAkYmIqgHYIp4/ambzEnZ+I2l/oBnYDNgI+HfKsZXjHEnHxPOehIwFHxSrnMxesPDxYZ69wHGclUeGI7VKybqjmwocmyyQtDbBOfzLzBZI+kDSTsAgQlYCCM7mWDObntd2D1pmFzgJ2ADob2ZLJb1J6ywGpcbWQh5M0taEJK8fSWqkZcTcEOsMJOhc7hXHP6qKPh3HcVYuGY7UKiXrU5ePA90kfRNAUkfgf4Hh8Z0YhOnLHwM9zCyXAPVh4OzcNKSkXYrY7wHMiU7uQGDLKsZ2K7CvpINjH12BKwnRJQSx6F3jvV2BXELVHsCH0cltR2H9S8dxnGxQw6lLSYdJmi7pjZgXNP9+D0n/kDQxrmE4pRaPkGlHZ2YGHAMcL+l14DXCe7Wk2OJdwAmEacwcFwGdgUlxa8BFRbq4FRggaRwhunu1irEtBI4Ezpf0GjCXsDjl1ljlbuBzkiYAZ8SxAzwEdJI0KY7rhUr7dBzHWdlY09KKj1LEQOUa4EvA9sCJkrbPq3YmMM3MdgYGEtZNpN5OlvWpS8zsHeCIEvf/Q95zRCf0nQJ1hwPDE9dzCYtTCtntHn++Scg0XqjOFMIiFiQdDfyfpNvM7K04hkOLDPtLRez1KlLfcRynfajdO7rdgTfMbAaApDuAo4Bpyd6AteJsXHdgHiG9Wioy7+hWFcxsJDCyzTuaMzu1ibTZB9Y45zepx7Do599NbaPxg0WpbXRcJ90fix02XDv1GI7dZLfUNjpsuXGq9vusn/6zbF6wOLWNtX52UmobS4aPSNV+7sT0uY+7NyxJbaNzQ0akt6p4Rxe3SyW3TA2Li+kgLPZ7J3FvFrBHnomrCfuT3wXWAgZZDVKcu6NzHMdxilOFn0muEC9AoRXp+avI/wuYQNhDvQ3wqKRn4rayFSbT7+gcx3GcdqZ2i1FmEVbM59icELklOQW4xwJvADOB7dI+gjs6x3EcpzhNjZUfpRkL9Ja0VVxgcgJhmjLJ28BBAJI2AvoAM9I+wirt6CRtJOk2STOipNboxEbscm1bpaeWdHpuK0MVY+gkaa6kS6pp5ziOs0pQo4jOzBoJMokPA68AfzOzqfF79/RY7SJgb0mTCdvLfhIXDaZilX1HF1fljARuNrOvxbItCUv+8+t2ih9ySczsuhUYyqHAdOCrkn4Wt0Tk99/R6iGpk+M4qx81VEYxsweAB/LKrkucv0vx1eorzKoc0X0RWJL3Ib1lZldBawHnSgzmhJglfUHSmER5r7jvrRAnAlcQQu49E23elHSBpGcJ+wAPjRHny3Fc3WO9C6L49BRJw6rQ2nQcx2l76kDrclV2dDsAL5epkxRwrhgzewVYI0p6QZAX+1t+vaiGchDwT+B2gtNLssjM9gUeA34OHBwFpMcB/xPrXG1mu5lZX6ArcHiBfpaLOj/+Uv5tx3GctsOzF2QHSddE2ZixieKkgHO1/I2YGYHg6AptzjkceDLKkd0NHBN3/+fItdmToATwXFRKOZnlcmMHKqQbmkyIUncgDzMbZmYDzGzAaQf1X8HHcRzHWQHqIKJbZd/RkSf4bGZnKmTnHpeo81mrVpUzArhT0j3BvL1eoM6JwD5RDBpgPYJSymN5/YvgdFtEfJIagGuBAWb2jqShuMCz4zhZovxqysyzKkd0TwANks5IlHWrlfGYW64J+AUFormYRWFfYAsz6xXlu86k9fQlBD3LfSR9PrbtJmlblju1ufGd3XG1Gr/jOE5N8Iiu/TAzi/qSl0v6MfA+IYL6SYUmukmalbj+vwJ1RgC/Z3nmgST/TUjumtQ9uhf4naQueWN9X9Jg4PbEvZ+b2WuSrgcmE7IdJKddHcdx2p/WC8lXOVZZRwdgZu8RNh0WujechIBzgftlo1kzuwy4rFL78X3gBvGyV969J4BWooZm9nPCQhXHcZzskeFIrVJWaUfnOI7jtDHu6JyVjc37IL2Rrl3K1ylBLTIPNPz62tQ2PjvztNQ2upz9g1TtF17y29RjeOiDt1PbaJyyoHylEry75KP0Y1iY/hX5x7+6LbWNJQs6lq9Ugh5rps/k0NSUfvnDv99NnxmjZ/kq5cnwtoFKcUfnOI7jFKdp1Rd1ckfnOI7jFMenLh3HcZy6pg4cXeb30aXJULAykLS7pKclTZf0qqQbJNVsP5/jOE67UgcSYJmO6KrJUFDCRptlDoj5ku4ETjCz0XG8xxJSwJddHVBpVgXHcZz2wppX/X10WY/oymUo6CXpmZgR4GVJe8fygZKelHQbYTM2kkbGiHCqpCE5e5JOk/SapFGSrpd0dSzfQNLdMbPAWEn7FBjfmQQnPDqOzczsLjP7T4z0npc0Pv7sE+22yKogaZMYEU6IGQz2a5NP0nEcZ0WoXeLVdiPTER3lMxTMAQ4xs0WSehMyCAyI93YH+prZzHh9qpnNixkHxkq6G+hCkPjaFfiUICs2Mda/ArjczJ6VtAUhWeAX8vrvC9xcZGyvAvubWaOkg4HfsFybcy9gpzieHwAPm9nFURC61bRndMxDAK46YSCn7ttK99lxHKdtqIOILuuOrgWSriHoSy4xs92AzsDVkvoRdCm3TVQfk3ByAOck3u31BHoDGwNP5TIcSLozYeNgYPtEeri1Ja1lZp9WONwewM3RAVsca45kVoWxwI2SOgMjzWxCviEzGwYMA1hwzVmr/m+d4zirDr4Ypc2ZSoi2gJChgJD/LSez9X3gP8DOhEhujUTbZZkLJA0kOK69zGxnYDxBULlUktMOsX6/eGxWwMlNBYrlzbmIkMKnL3AELbMSLBubmT0N7A/MBv4i6ZslxuQ4jrNyqQNR56w7unIZCnoA75lZM/ANoJgkQg/gQzNbIGk7lmcCHwMcIGldSZ1IpP0hZCU/K3cRo8Z8rgZOlrRHot7XJW0c+5wdiwcXe8C4uGaOmV0P/JmEY3ccx2l3zCo/MkqmHZ2ZGXA0wRnNlDSG8E4sl6HgWoKjeYEw5Vgs/9xDQCdJkwiR1gvR/mzCu7MXCTnkpgEfxzbnAAMkTZI0DTi9wPj+QxCVvixuL3gF2A/4BPgdcImk5yjugAEGAhMkjSc42itKfyqO4zgrkRpGdJIOi9+Vb0g6r0idgXFx3lRJT9XiETL/jq5MhoLXgZ0SRT+N5aOAUYl6i4EvFeniNjMbFiO6vxMiOcxsLiGzeLnxjSY4t3xG0/Kd4S9i/eEksh6Y2c0UX9DiOI7TvtRIAiwutrsGOASYRVgUeJ+ZTUvUWYcQwBxmZm9L2rAWfWfe0a0EhsZVkQ0EJzeyncdTmhrMg9sn81O1b/wgvehtLQSZ17zmz6ltLPxpq0C9KrqccnzqMWz55F9S2/h4Qrovoy26rJd6DB27pBOWBpj3TnqthTV7LC5fqQQ9j0j/tfjG3alN0LlTRjQma7fqcnfgDTObASDpDuAowkxajq8B95jZ2wBmNqcWHa/2js7MftjeY3Acx8kqVsUf18mtUJFhcdU4wGbAO4l7s4A9aMm2QGdJowjCG1eY2S3Vjjmf1d7ROY7jOCWoIqJLboUqQKFV7vnGOxFWsh8EdAVGS3rBzF6reBAFcEfnOI7jFKd2GpazaJkib3Pg3QJ15prZZ8Bnkp4mbB9L5egys+pS0npxpc0ESf+WNDtxvUZe3XMrEU6Osl4DipRPj7ZfSUqCpRj/OpKKZiSVdH5cRTQp9rtHgbFMkHRc2rE4juPUjMamyo/SjAV6S9oqfqefANyXV+deYD9JneJ3/B7AK2kfITMRnZl9APQDkDQUmG9mlxWpfi7wVyoQTi7BSWY2TtLngH9JGm5mS1LYWwf4LmHFUAsk7QUcDuxqZoslrU/Lze0nmdm4FH07juO0DTVajBLlEM8iyCl2BG40s6mSTo/3rzOzVyQ9BEwCmoEbzGxK2r4z4+gKIekg4DLCOMcCZwDfATYFnpQ018wOlPRHYDfCnO5dZvbLKrrpTth/1xSXv/6ZoLJihH+Iy+OL0fGEueMNgG8StjLsCIwws58DvwW2kTSBIPH1o0QfmxDC8cWwbOuC4zhO9qlh+h0zewB4IK/surzr3wO/r1mnZGjqsgANhP1mg8xsR4KzO8PMriTM6x5oZgfGuueb2QDCnroDJO1UyGAet8YN5NOBi2Iqn37AZmbWN/Z5U6L+EjPbH7iOEF6fSRB1HixpPeA84F9RLuxHtOQRoKdCloRrJR1QYCy5qctW67wlDZE0TtK4G5+bln/bcRyn7Wi2yo+MkmVH1xGYmVhtczNBE7IQX5X0MiHq2gHYvgL7J5nZTsAWwA+jFNcMYGtJV0k6jKBwkiM3lzwZmGpm78UIbQYtX7C2wszmE6LBIcD7wAhJg/PGktPU/KBA+2FmNsDMBpy6TyWP5jiOUxusubniI6tk2dEVk/NqgaStgB8CB0XHdT8tBZRLYmbvE1IB7WFmHxJW+IwiRGw3JKrmdqE2J85z12WngM2sycxGxWnVs2ipq+k4jpNNPKJrUxqAXpI+H6+/AeR0zz4lbCYEWJvgFD9WyPhdTOqrIHFlzy6EBSnrAx3M7G6W56mrlOSY8vvoE9P15OgHvFXNOB3HcdqFpqbKj4yS5cUoi4BTgDujDuVYwvsxCBsSH5T0XlyMMp6QMmcG8FyF9m+VtJCQfHW4mb0kaWfgJkm5PwB+WulgzewDSc9JmgI8mPeerjtwVdRxawTeoKV6gOM4TjbJcKRWKZl0dGY2NHG5S4H7VwFXJa4HF7EzsMryiRSI4pL1CwhGJ+99rYjdl4C9qxmL4zhOFjB3dI7jOE5d447OWRVRp1Lp8crTcZ01ylcqQ5ezf5DaRtrMAwBdL7mufKVSYzj/jPKVyjBvSX7i+urptnG6f5N5s5emHkNzYyEpw+rotlYazYbAkgXpvtbmPrEw9Rg6dqh4PVxRmprTf541IcOrKSvFHZ3jOI5THI/oHMdxnHrGmjyicxzHceqZOojosryPDgBJG0m6TdIMSS9JGi3pmPYeVxJJV8RsC5n/PB3HcarCN4y3LZIEjASeNrOtzaw/IbXD5lXYSLfyorz9DsAxhMy5BSXK4j5Ax3GcVQ5rtoqPrJJpRwd8kSCmvGxpnJm9FffRIamXpGckvRyPvWP5QElPSrqNoE2JpJExIpyazD8n6bQotjxK0vWSro7lG0i6W9LYeOxTZIwHAlOAPwInJuwOlTRM0iPALcXsSdpd0vOSxseffWr4+TmO46SjDiK6rEcaOxB0KIsxBzjEzBZFia3bCSl2AHYH+prZzHh9qpnNk9QVGCvpboIqSk7q61PgCWBirH8FcLmZPStpC0IOpS8UGMOJsd97gd9I6mxmubXa/YF9zWxhdLqF7L0K7B9zNR0M/IY8HczomIcAXDXoAFzY2XGclYU1ZteBVUrWHV0LJF0D7EuI8nYDOgNXS+oHNAHbJqqPSTg5gHMS7/Z6Ar2BjYGnzGxetH9nwsbBwPZh9hSAtSWtZWbLNj3FLLlfBr5vZp9KehE4lCAsDXCfmS0sZQ/oAdwcHbXFZ2qBmQ0jyJ6x4Krvrvq/dY7jrDpkOFKrlKw7uqkkohszOzMKL+eycX8f+A8h40AHgj5mjmXZDyQNJDiavcxsQUyk2gCU2pHZIdYvtXv0MIKjmhwdWDdC1vOco0tmYChoT9JVwJNmdoykXiTkxRzHcdqdVX93Qebf0T0BNEhKyk90S5z3AN4zs2ZCdoNiC096AB9GJ7cdsGcsH0NI1LpuXDCSnDJ8hJBOB4AYNeZzIvAtM+tlZr2ArYBDY0aEfIrZ6wHMjueDi4zfcRynXfDFKG2MmRlwNMEZzZQ0hpCA9SexyrXAyZJeIEw5Fsth9xDQKWYUvwh4IdqfTXgn9iLwGDAN+Di2OQcYIGmSpGlAC72p6Mz+i+XRG2b2GfAscESBMRSz9zvgEknPUdxRO47jtA/NVRxlkHSYpOmS3pB0Xol6u0lqknRc+gfI/tQlZvYeYUtBoXuvAzslin4ay0fRMsPAYornqbvNzIbFiO7vhMgLM5sLDCoxrgXA5wqU/3eR+gXtmdloWr5b/EWxPh3HcVY2tYrU4lava4BDgFmERYH3mdm0AvUuJSzYqwmZd3QrgaFxtWMDwcmNbOfxlKYGAqtN736Yqn2HDddOPYaFl/w2tY0upxyffhwpRZm7XvzH1GPY7oH0wtBNCxaVr1SC+78gznxjnVQ21OGjVO0BGtZpTG2jW0M6Gx27p5/o+vC59DYsIzOBlv6fJMfuwBtmNgNA0h3AUYSZtCRnA3cDu9Wq49Xe0ZnZD9t7DI7T3qR1ck4dU8Xf1smtUJFhcdU4wGYEYY0cs4A98tpvRhDg+CLu6BzHcZyVgVXh6JJboQpQaJV7ftz6B+AnZtaU2IqVGnd0juM4TnFqt71gFmEPc47NgXfz6gwA7ohObn3gy5IazSzVKyV3dI7jOE5RqonoyjAW6C1pK8KWqhOAr7Xoy2yr3Lmk4cA/0zo5yPj2glog6UZJcyRNqaDuwJxeZoF7gyW9L2lCPG6J5cMrWQIrqU/U05wg6RVJwxJ9fpyw+1i1z+g4jtNWWHPlR0k7Zo2EvcQPA68AfzOzqZJOl3R66dbpWB0iuuHA1cAtFdQdCMwHni9yf4SZnVXkXgskdTSzpkTRlQSty3vj/R0T954xs8Mrses4jrMysabavSszsweAB/LKritSd3Ct+q37iM7Mngbm5ZdLOkfStLiB+44ov3U68P0YWe1XbV+S3pR0gaRngfy175sQ5qhz45pcrX3HcZyVTa0iuvak7h1dCc4DdjGznYDTzexN4DpC1NXPzJ4p0GZQYorxlCJ2F5nZvmZ2R1755cATkh6U9H1JyfXc+yXsnp9vUNIQSeMkjbvx+fwtJ47jOG2HNaviI6usDlOXxZgE3CppJJVvEq9k6nJEoUIzu0nSwwQh6KOA70jaOd4uOXXZInvBFadnZBup4zirA1mO1CpldY7ovkKQo+kPvKTaZQEvpreJmb1rZjea2VFAI9C3Rn06juO0CWaq+Mgqq6Wjk9QB6GlmTwI/BtYBuhOSr67VRn0eJqlzPN8YWI/lWQscx3EySXOjKj6ySt07Okm3A6OBPpJmSTqNkCXgr5ImA+MJ7+U+Av4BHLOii1HKcCgwRdJEwvLaH5nZv2vch+M4Tk0xq/zIKnX/js7MTixya98CdV+jZTaE5L3hhK0K+eWDE+e9Sozjf4D/KVA+Ck+26jhORsnyIpNKqXtHV2+sd96DqW0sbUonR37sJum1Vh/64O3UNrZ88i+pbcxb8mmq9rXIPPDg+PQZED7f5+hU7V85b5vUYzj6ysWpbezScd3UNj6hqXylEsxqXpB6DGsp/WRZQ4f06SlvTG3BHZ3jOI5T52R5SrJS3NE5juM4RfGIznEcx6lrmmsoAdZepJpIVuBZSV9KlH1V0kPph1ay3+GSZsbVka9K+mUFbbaL9cdLSv9CoqXtUyVNjnJiUyQdVab+UEme8NVxnMzTbKr4yCqpIjozs6g6faekJwnL9i8mqH9UTQEh5FL8yMzuktQATJN0i5nNLFH/aOBeMyvrFONYBMistC6ApM2B84FdzexjSd2BDSp8BsdxnEyT5Y3glZJ6aZCZTSHsP/sJ8Evgr8D5ksbG6OkoAEm9JD0j6eV47B3LB0p6UtJtwGRJa0q6X9LEGB0NKjOEhvjzs2ivv6SnJL0k6WFJm0j6MnAu8K3okJH0P9H+FEnnJsb4iqRrgZeBnpJ+FJ9lkqRfFeh/Q8JG8/nx85ifc7iSvh3bTpR0t6Ru+Y0lbSPpoTjeZyRtV+FH7ziO0+bUg9ZlrTaM/4qQQO9LBMfzhJntBhwI/F7SmsAc4BAz2xUYREhbk2N34Hwz254QDb5rZjubWV+g2DTo7yVNIGQEuMPM5kTlkauA48ysP2F17cUxNUROsPlASf2BU4A9gD2Bb0vaJdrtA9xiZrvE895xfP2A/pL2zxvHROA/wExJN0k6InHvHjPbzcx2JuRfOq3AcwwDzo7j/SFwbX6FpKhzY+P8Ih+H4zhO7fEN4xEz+0zSCEJU81XgiMQ7qAZgC0LK9Ksl9QOagG0TJsYkph0nA5dJupSQXbZQFgFYPnXZHXg8RoifEPQjHw0zj3QE3ivQdl/g72aWiwLvAfYD7gPeMrMXYr1D4zE+XncnOL6nE8/eJOkwYDfgIOBySf3NbCjQV9KvWS4x9nByEHHsexOmfnPFXfIHmxR17tp1ywz/OjmOU29kOVKrlFquumyOh4BjzWx68qakoYTIZ2dCJLkocXuZELKZvRYjri8Dl0h6xMwuLNapmc2XNIrgvB4EpprZXmXGWupfLinKLOASM/tTKWNmZsAYYIykR4GbgKEEJZWjzWyipMGExK5JOgAfmVm/MuN1HMdpF5qaV32lyLZ4goeBs+NiDhJTgj2A9+Lijm8Qoq1WSNoUWGBmfwUuA3Yt1VnMOrAH8C9gOrCBpL3ivc6SdijQ7GngaEnd4rTqMUChyPFh4NQYeSFpM0kb5o9XUnKM/YC34vlawHtxSvWkfONm9glhyvP4aEtanrrHcRyn3amHqcu2cHQXAZ2BSZKmxGsI755OlvQCYdqyWDqbHQmR0QTCasZfF6mXe0c3iTDdeY+ZLQGOAy5VEE+eQJgabIGZvUyItsYALwI3mNn4AvUeAW4DRisIQN9F6+wGnQlTra/G8QwCvhfv/SLafxR4tchznAScFsc7lZCrznEcJxPUcnuBQhaX6ZLekHRegfsnxYV/kyQ9X6s//GVZdsNOK2rxji4bWpeTU9vYsvuG5SuVIbXW5ZqbpR5DNrQuB6Qew9FXvpvaRv1oXXZObaNBNdC6fPOu1P6dmTIAACAASURBVC/Yxm9xVMXfObu8fW/R/iR1BF4DDiEsIhwLnGhm0xJ19gZeMbMPFfZnDzWzPVZ48BFXRlnFeP/c9F9ILF6aqnmHLTdOPYTGKem/TD6ekO4LDaDbxmukat+0YFH5SmVI66QA3pg+MrWNPtsdm6r9/etuknoMPY9K/3sx466UKbE7iOmL105lYjPS/16s2z39Z1ELahgL7Q68YWYzACTdQZjBWubozOz5RP0XgM1r0bE7OsdxUju5eiKtk6s3qlmMImkIMCRRNCyuGgfYDHgncW8WYX1FMU4jLDBMjTs6x3EcpyjVSHslt0IVoJChgvGipAMJjq5V3tAVwR2d4ziOU5QaruKYBfRMXG9O2F/dAkk7ATcAXzKzD2rRsTs6x3Ecpyg1FGseC/SWtBUwGziBoKi1DElbAPcA3zCz12rV8aq/E7AKJDUpZDDIHb1K1B0s6ep4XjDbQCyfreVZFP4olU4tLOloSdsnrkdJqsEKE8dxnNpjpoqP0nasETiLsD/5FeBvZjZV0ukKyQEALgDWA66N36vjavEMq1tEt7ANVEguN7PLooN7GjgAeLJE/aOBf5JYaeQ4jpNVUq5hbUHUHX4gr+y6xPm3gG/VsEtgNYvoCiHpTUnrx/MBUU5sRViDoOv5YbTVKnNB3CNyJHGzu5bnxTte0hhJr0naL+UjOY7j1IwmU8VHVlndHF3XxLTl32tk8/tREeU94DUzmxDLW2UuiHtE7iMIUvczs3/Fup3MbHdCKqFW+fKS2QtunFAq5Z7jOE5taUYVH1lldXN0C6OD6Wdmx9TI5uVxOnRDYE1JJ8Tyvgr55SYTZL4KaW7muCf+fAnolX/TzIaZ2QAzG3Bqv61qNGzHcZzyGKr4yCqrm6MrRCPLP4eGUhVLYWZLCbnzcvnqhgNnmdmOhHx9pWwvjj+bWP3emzqOk2Gaqziyijs6eBPoH89XWB4iZmvYm5BFAYpnLviU1sLQjuM4mcQjuvrgV8AVkp6BFVKDzb2jm0KIxnIZwotlLrgD+JGk8YnFKI7jOJmksYojq6xW02Rm1r1A2TO0zHaeKx9OmH4kZgsvZG8oIcFqoXt/BFrJ0pvZc8D2iaKBiXtzKfCOznEcp73IcqRWKauVo6sHNrtqQvlKZVjcmC57wT7rp1dmf3fJR6ltbNFlvdQ25s1O91n06pw+rcwr56UP7NOKMk9/9e7UYzhjwI9T22i6K73g1LqkS5HT2CV9bDKvBpL/i5ekT/VzZ2oL0Lzq+zl3dI7jOE5xsrxtoFLc0TmO4zhFqYfU3O7oHMdxnKJkedtApZQTIJakZ2NK81zZVyU91JaDkrSnpBejgskrkobG8iMlndeG/Q6W9H5CpPn7bdVXkf5d4NlxnEzRJFV8ZJWSEZ2ZWVSVvlPSk0BH4GLgsBXpTFJHM6tkCf/NwFfNbKKkjkCfOJ77CBJabckIMztL0nrAdEl3mdk7ZVs5juPUIXUf0QGY2RTgH8BPCDqMfwXOj4LF4yUdBSCpV5S8ejkee8fygZKelHQbMFnSmpLuj2LHUyQNKtDthgTtSMysycymRVvJ1DnDJV0p6XlJMyQdl2ss6ceSJsc+fhvLtpH0kKSX4ji3K/PcHwBvAJvE9l+PwssTJP0pOmAkzZd0abT7mKTdY2Q2Q9KRsU6DpJvimMYrZM9FUldJd0iaJGkE0LXcv4fjOM7KpFmVH1ml0nd0vwJeBpYQUsw8YWanSloHGCPpMWAOcIiZLZLUG7gdyE3D7Q70NbOZko4F3jWzrwBI6lGgv8sJ0dQogqzWzWZWaE37JoRU69sRIr274jTr0cAeZrZA0udi3WHA6Wb2uqQ9CBu7v1jsgRUSADYAkyR9ARgE7GNmSyVdS1A7uQVYExhlZj+JQtG/Bg4h7JW7OY7rTAAz2zE62EckbQucASwws50Usuq+XGQsQ4AhAA1rrM8andcuNmzHcZyastqsujSzz2LEMR/4KnCElicibQC2IKREv1pSP4LCSHIT9hgzy8nuTwYuk3Qp8M+4YTu/vwsl3QocSshAeyKJjdUJRppZMzBN0kax7GDgJjNbEG3Nk9SdIM91p5bPI3cp8riDYsTVB/h2dNwHEWTCxsb2XQmOHYLzz72znAwsjs5wMss3f+8LXBXH86qkt+Lnsz9wZSyfJGlSoQGZ2TCCo6ZH923qYRGU4zirCPXwhVPNqsucbqeAY81sevJmXDDyH2BnwpRoMgL7LHdiZq9J6g98GbhE0iNmdmF+ZzGFzR8lXQ+8H9+Z5bM4ca7Ez/x/mw7ARxUmXc29o9sLuF/Sg9HmzWb20wL1l5ot2x3anBuTmTVLyn2+pf4kqoffI8dx6pQsT0lWyopoXT4MnB1FjJG0SyzvAbwXI6xvEBautELSpoTpur8ClwG7FqjzFS0PvXoTIsRKpTQeAU6V1C3a+pyZfQLMlHR8LJOknUsZMbPRwF+A7wGPA8dJ2jBnU9KWFY4HQubxk2LbbQkR8PS88r7ATlXYdBzHaXOaqjiyyoo4uouAzoR3V1PiNYR3XidLeoEwLfdZkfY7Et7rTQDOJ7zTyucbhHd0EwjO5qQKV2tiZg8R3ouNi+1zU6wnAadJmghMBY6qwNylwCnAO8DPCe/WJhGEmjepZDyRa4GOcTpzBDDYzBYTtDC7R5s/BsZUYdNxHKfNqeViFEmHSZou6Q0V2CoWg5Ar4/1JkloFQiuCrAaabM7Koxbv6NJrXZZcsFoRmdG6bCr291hl1ELr8uaz0j9H30vHp2qfGa3LGszkp9a6rMEY5lm6/8cAFtcgRrrzrXtTTzxev/nXK/5Avj3rr0X7iyvVXyMs1psFjAVOzK2qj3W+DJxNeLW1B3CFme2xgkNfhqfpcRzHcYpSw8SruwNvmNkMM1tCSFmWP7N2FHCLBV4A1pFUzexZQVwCbBVj1uA+qW0snZ0uimlesLh8pTI0LuyW2kbHLgtS22huTPcHrzqkj0yPvjL953n/uum+C2oRjf1x3O9S25ix75mpbaSlU6f0W6Rnz02/BWiBrZHaRi2wKv4XSW6FigyLq8YBNiO8BsoxixC1JSlUZzPivuoVxR2d4ziOU5RqkhYlt0IVoJDLzJ8WraRO1bijcxzHcYpSw1Ucs4CeievNCfuvq61TNf6OznEcxylKDVddjgV6S9pK0hrACbTWLr4P+GZcfbkn8LGZpZq2hAw6Okkm6S+J604KGQX+mdLucEnfySs7WtIDaexGO/dKGp3WjuM4Ttao1WIUM2sEziLsxX4F+JuZTZV0ukLyAIAHgBkEneHrge/W4hmyOHX5GdBXUlczW0hYijq7BnZvB84D/pQoOyGWV4SkTvEfK1m2DmHT+3xJWyWkzkq2cxzHWRWoZfYCM3uA4MySZdclzo2oDVxLMhfRRR4EvhLPTyThjGJ2gOdjFoDnJfWJ5TtoeXaBSVFYOsljwHa5papROeVgYKRC5oVXJF0vaaqkRyR1jfVGSfqNpKcIKin5HEvI7nAHwXHmxjlc0v8ppDe6VEWyJ0g6QiH33niF7AcbFejDcRynXbAqjqySVUd3B3CCpAaCLNaLiXuvAvub2S7ABcBvYvnphM2F/QhZE2YlDUZllXsIotQARwJPmtmn8bo3cI2Z7UCQGzs20XwdMzvAzP63wFhzjvj2eJ5kW+BgM/sBYSXS2WbWn6DWcm2s8yywZ3yeOwgKKS2QNETSOEnjbpr6doEhOI7jtA2NqvzIKlmcuswp+fciOI78d2g9gJtjxGawTAZhNCFP3ubAPWb2egHTtwO/B64gRF+3JO7NNLMJ8fwllmcegCDb1YoYfX0eeDYmqW2U1Dfm8AO408yaVDp7wubAiBhprgG0mvpMLtn99KwvZ/kPJ8dx6ox6+MLJakQHYfXNZbR+h3YRIRLrCxxBSBOEmd1GiNIWAg9LKpRr7jlgkyjovDctnWhy124TLf8IKLbDehCwLkEw+k2CczwhcT/Xbln2hMTxhXjvKuBqM9sR+E7ueRzHcbJAM1bxkVWy7OhuBC40s8l55T1YvjhlcK5Q0tbADDO7kuAkW2UCiC86/0ZIiPpAkWSu1XAicJiZ9TKzXoScdSfkVyqTPSH5PCenHI/jOE5NqaEEWLuRWUdnZrPM7IoCt35HyGP3HC1TAQ0CpsSMBdvRcloyye2EnHl3pBlfnFrdAnghMeaZwCcKGczzKZY9YShhSvMZYG6aMTmO49SaeliMkrl3dGbWvUDZKGBUPB9Ny+zlv4jllwCXVGB/PHkyM2b2JtA3cX1Z4nxgETtvEjTY8stzaSVezCufCRxWoP69wL3lxu04jtMeZDlSq5TMOTqnDM3p/25q/CRd+7V+dlLqMXz8q9tS25j3Tnph6G5rLUnVvmGd9Nsjd+mYPtVPz6PSCVw33ZX+96oWgsxbP3tNahuzDvpO+UolGDU3/Q6fLS3d7xXAxg3pRctrQaOyHKtVhjs6x3EcpyirvptzR+c4juOUwKcuHcdxnLomy9sGKmWlrLqUtF6U5pog6d+SZieu18ire26U58pdvylpcpT1ekrSlm00xmOioPR2ibKB1YpJS9pI0m2SZkS5r9GSjqmw7ShJA6odu+M4TltRD6suV4qjM7MPchulgeuAyxMbp/Pf2p4L5K8yONDMdiKsvPx5Gw3zRIIcV6t9cJWiIHsyEnjazLaOcl8nENRP8ut6NO04TuZpxCo+skq77aOTdFAUMp4s6UZJXSSdA2wKPBnFkPMZTVzSH4WYX5V0g6Qpkm6VdLCk5yS9Lmn3WO+ARPQ4XtJaBcbSHdgHOI3Wjm5tSX+XNE3SdZI6SDpD0u8S7QdLugr4IrAkT437LTO7KlHvTkn/AB6R1FXSHTFaHQF0XfFP1HEcp/Z4RLfiNADDgUFR+qoTcEZUNXmXEMEdWKDdYYSIKcfnCbqVOxE2iX8N2JcgmvyzWOeHwJkxmtyPIBGWz9HAQ2b2GjBP0q6Je7sDPwB2BLYB/hu4K/7MMYigh7kD8HKZZ98LONnMvgicASyI0erFBGUVx3GczODKKCtOR4KI8mvx+mZg/xL1n5Q0h5BWJ7kBa6aZTTazZoLayONR5msyy0WZnwP+L0aL6xTJC3ciy5VS7qBlFoIxZjYjZj+4HdjXzN4HZkjaU9J6QJ/YTwskXSNpoqSxieJHzWxePN8f+CsEIWtgUqGHb5G9YNo7hao4juO0CVbFf1mlvRxdMZHkYhwIbElwZhcmypNCzM2J62biilIz+y3wLcK04AvJxSYQFsoQphxviMLMPwIGaXmagfx/vdz1CELKn2OBv0cHO5WQhJXY95nAQcAGifb5z172t8PMhpnZADMbcMr2PctVdxzHqRke0a04DUAvSZ+P198AnornnwKt3qPFbOPnAt+U9LlKO5K0TYz6LgXGEaY4kxwH3GJmW0Zx5p6EVDn7xvu7S9pKUgfCFOWzsfwewpTniSxP4/ME0CDpjIT9UvIdTxM0MJHUlwJC1I7jOO2JZy9YcRYBpxDEjCcT/hjILeAYBjxYaDGKmb1HmD6sRmvo3LhYZSLh/dyDefdPBP6eV3Y34X0fhAUwvwWmEBzg3+NYPgSmAVua2ZhYZgTnd4CkmZLGEKZlf1JkbH8EukuaREi4OqaK53Icx2lzmrCKj6yy0pe4m9nQxOUuBe5fRcjRlrvulXf/7MRlUoh5cOL8zdy9vPqFxjOwQNmVictRJdoeXqDsPYpsUTCz4YRFOLnrhcXqOo7jZIGVNSUZZ+pGENZXvAl8NQYUyTo9CZlpNo5DG1Yky00LMpumx3Ecx2l/VuJilPMICwp7A4/H63wagR/ExNV7AmdK2r6cYd+0vIphi9Or5XdYo3ydUiwZPqJ8pXI2FnQsX6kMa/ZYXL5S2XGk+1+gW0P6f49PaEptY8Zd6f7uXpfOqcdQC9JmHgDY/PE/pWp/wIGnpx7DJ580pLbx0aIuqW3UgpW4yOQoYGA8v5kwm9bitU+cMXsvnn8q6RXC3upppQx7ROc4juMUpZqILrkVKh5Dquhqo+jIcg5tw1KVY/LrXcjL/VkIj+gcx3GcolQT0ZnZMMKCwoJIeozwfi2f86sZU1Szuhs418zKZth0R+c4juMUpclqt5rSzA4udk/SfyRtYmbvSdoEmFOkXmeCk7vVzO6ppN+6m7qU1JTQtpwQNTGfr6H9NyWtXyt7juM4WWYl7qO7Dzg5np8M3JtfIQp5/Bl4xcz+r1LDdefogIWJzAj9zOxNM9s7v5Kk9KshHMdx6pyVuOryt8Ahkl4HDonXSNpU0gOxzj4EgZEvJoKZL5czvFpMXUqab2bdJQ0EfklYtdNP0o6ED3Mg0AW4xsz+FOtdCHxA0LF8Gvhu1NRM2h0J9CQovVwR56eRdBjwG4Km51wzO0jSmoT9gTkR66Fmdq+kHYCbgDUIf3gca2avt9mH4TiOUwUra9WlmX1AkEzML38X+HI8fxZQfp1y1KOj6yppQjyfaWb5SU93B/qa2cy4IuhjM9tNUhfgOUmPJOptD7wFPMTyrAVJTjWzeZK6AmMl3U1wVtcD+8c+cnJl5wNPmNmpktYBxsQXs6cTnOStCkloW0WacZxDAP6w93acsl2r9HaO4zhtQpalvSqlHh3dwpiSpxhjzGxmPD8U2EnScfG6B9AbWBLrzQCQdDtB+zLf0Z2TyB7eM7bdgJB4dSZAIlPBocCRkn4YrxuALQgSY+dL2hy4p1A0l1zJ9Mlph6z6v3WO46wyZFnaq1Lq0dGVI5k9QMDZZvZwskKcuiyWtSBZ52BgLzNbIGkUwXmpQNtcX8ea2fS88lckvQh8BXhY0rfM7ImqnshxHKeNsBquumwv6nExSjU8DJwRl6siadv4Lg2KZy3I0QP4MDq57QhyNBAitAMkbRVt5qYuHwbOzqX/kbRL/Lk1MCPqa96HZzBwHCdD1EP2gtUxoktyA0FA9OXogN4nZB+A5VkLdiQsRsnPcPAQcHrMPDAdeAHAzN6P79TuiU5yDmEF0UXAH4BJsa83gcMJTvTrkpYC/6Zlvj3HcZx2Jct55iql7hydmXUvVmZmo0hkI4irKH8Wj2XEoGuBmQ0qYKtX4vJLRcbwIHnpgGKmglZCfmZ2CXBJ4adxHMdpX7KcObxS6s7R1TvNC9KLCH/0VjrB2bkT1yxfqQw91lyU2kbPI9L/+s59YmGq9h27p5/9n9W8ILWN6Ut7pGq/F/BMl3S/W506pf/bf9TcjVLbSCvK3PPJ68pXKsOUXb+f2sanysbXc5anJCslG59kxsiP/Byn3knr5Jz6pZYSYO2FOzrHcRynKD516TiO49Q1PnXpOI7j1DW+j64NSWQhmCjpZUmthJlXwGa/pACopMGS3k+Ig94i6UhJhVK4l7J7vqSpkiZFO3vE8lGSpifsH1eg7cWS3pE0P+3zOY7j1BrfR9e2LJPykvRfhCX4B6S02Q8YADyQKBthZmfl1buvUoOS9iLsh9vVzBbHFD5rJKqcZGbjSpj4B3A14ELOjuNkjiZb9XfSZTaiy2Nt4EMASZtIejpGSFMk7RfL50u6VNJLkh6TtHuMqGbEKG0NwmbsQbFtqz1y0c5gSVfH8+GSrpT0fLTTKiIDNiFkKFgMYGZzo9p2RZjZC7n08Y7jOFnDqjiySpYdXdfokF4lKJhcFMu/Bjwco72dgVymgjWBUWbWH/gU+DVBkeQY4EIzWwJcQIjg+pnZiNgu5/gmSDqlwDg2IQg6H07Mj5THI0BPSa9JulZSftR5a8L+eivwOSBpiKRxksYNf6NiH+o4jpMan7psW5JTl3sBt0jqC4wFboz6lCPNLOfolhBkuQAmA4vNbKmkyQSZr2K0mLqUNDjv/siooDJNUqvdrGY2X1J/YD/gQGCEpPPMbHisUm7qsizJ7AUfnXhgdn+bHMepO7LswColyxHdMsxsNLA+sIGZPQ3sD8wG/iLpm7HaUlu+PKgZyE0lNpPOoS9OnBdM+GdmTWY2ysx+CZwFHJuiP8dxnMxgZhUfWSXLEd0yYnaAjsAHkrYEZpvZ9THTwK7ALRWa+hRYq8Zj6wM0J/LI9SMka3Ucx1nl8Yiubcm9o5sAjABONrMmYCAwQdJ4QuR0RRU2nwS2L7UYZQXoDtwsaVrMZLA9MLTSxpJ+J2kW0E3SLEkVt3Ucx2lrmq254iMNkj4n6VFJr8ef65ao21HSeEn/rMR2ZiM6M+tYpPxm4OYC5d0T50ML3YvZvnfLazo8r+7wXJmZDS7WR6LsJaDgHj8zG1ioPK/Oj4Efl6vnOI7THqzEiO484HEz+23cy3we8JMidb8HvEJYkV+WzDo6pzDzZxR8TVgVS5cW/BuiYro3LEk9hqam9JMJb9yd2gQdO6TL5PDhc+mfYy2lt7EZ6bJBzKvB+5XZcyv6zinJlpb+d+uTT9L9m9Yi80Dfly9PbWPCzhektlELVuK7t6MIM3YQgplRFHB0kjYHvgJcDPxPJYazPHXpOI7jtDPVbC9IboWKx5Aqutoot6c4/tywSL0/EGbBKp4r9YjOcRzHKUo12QuSW6EKIekxYOMCt86vxL6kw4E5ZvaSpIGVjssdneM4jlOU5hpOXZrZwcXuSfqPpE3M7D1JmwBzClTbBzgyahY3AGtL+quZfb1Uv+02dSnpcknnJq4flnRD4vp/JVU0/1qij4GFVuXE8o/jqp3pUVLs8BT9lBVkTiP87DiO0140WXPFR0ruA06O5ycD9+ZXMLOfmtnmZtYLOAF4opyTg/aN6J4Hjgf+IKkDYUN48m323sC5hRrWiGfM7HAIWQ2AkZIWmtnjte6oBsLPjuM47cJKTLz6W+Bvkk4D3ib4ByRtCtxgZl8u1bgU7bkY5TmWL8vfAZgCfCppXUldgC8A4yUdFCOvyZJujPcoUX6YpFclPQv8dyUDiTJiFxJUTZC0gaS7JY2Nxz6xvLukm2KfkyS1UECRtL6k0ZK+ktdFKuFnx3Gc9qLZrOIjDWb2gZkdZGa94895sfzdQk4uqlFVNBPXbo4uftE3StqC4PBGAy8CexFS6UyK4xsODDKzHQkR6BmSGkqUXw8cQdCeLPTSsxgvA9vF8yuAy81sN8Km9NyU6i+Aj81sRzPbCXgi1zjqYN4PXGBm9+fZTiX8nFzJdOsc94+O46w8rIr/skp7by/IRXU5Rzc6cf080AeYaWavxfo3E3Qui5VvF8tfj7qXf61iLMkNagcDV0dVlvsILzzXiuXX5CqZ2YfxtDPwOPBjM3s037CZzQf6A0OA9wnCz4MTVU6KGRX6mdkHBdoPM7MBZjbgpA03reKRHMdx0rGyIrq2pL1XXT5PcGo7EqYu3wF+AHwC3EgREeUS5bDiaZF2Iey0h/AHwF5mtrBFp5KK2G8EXgL+C3iq4KCCfNkoYFTMqHAyeaosjuM4WaPZmtp7CKnJQkR3ODAvZgCYB6xDmL4cDbwK9JL0+Vj/GwRHUqp8K0nbxPITKxmEpJ0I05K5aO0R4vu6eL9fkfKcFpsBpwLbRemafPt9JPVOFLnws+M4qwT1kI+uvR3dZMJqyxfyyj6OCzYWAacAd8YoqBm4rkz5EOD+uBillDPZL7e9gODgzkmsuDwHGBAXnEwDTo/lvwbWVchsPpGQfw5YFrGdABwo6bt5faUSfnYcx2kvPE1PSqJzWDuvbHDe9eOEacX8tsXKH2L5opJi/Y4CepS4Pxdold0gvms7uUB5TjR6CWH6Mv9+KuFnx3Gc9iLLkVqltPc7OsdxHCfDZDlSqxR3dKsYzU3psxd0aWhM1b5zQ/qX0/9+N73SfedO6cfR1Jzu86zFd0BDh3TZJADW7b4gVfvFSzqnHsMCW6N8pTJs3JDuOQA+WtQlVftPlf5rsRaZB74+8cLUNmpBlldTVoo7OsdxHKcoaROqZgF3dI7jOE5R/B2d4ziOU9fUwzu69sxesLmkeyW9Lulfkq6QlH6Sv3SfgyW9H6W2pkn6dg1sDpX0wwrqnZrQyJwi6ahYPlzSzIQE2Dlpx+Q4jlMr6kEZpV0cXVQYuQcYaWa9gW0Je80uXgndjzCzfoSU7b+JGpVlkbTCKwYUUr+fD+wbNTL3JGh55vhRQgLsyhXtx3Ecp9bUwz669orovggsMrObYNl+uu8Dp0rqFiOveyU9FHO1/TLXUNLXJY2J0c+fcg5I0nxJF0uaKOmFcg7MzOYA/wK2LJEJ4U1JF8TN58fHzAgvxz6S6Xy2V8grN6NIRLYh8CkwP/Y938xmruiH5ziOs7JwZZQVZweCNuQyzOwTQg6inKzX7sBJBLms4yUNkPQFwkbufWJU1hTrAKwJvGBmOwNPAyWnJSVtDWwNzKJAJoRE1UVmti9BtPl64NjYx/GJOtsRNorvDvxSUv5a7YnAf4CZMc3PEXn3f5+YutyxwFiXZS+47f3ZpR7LcRynpjQ1N1d8ZJX2WoxSTBw5Wf5oTslf0j3AvgTx5P7A2DD7SVeWp1tfAuSyib8EHFKk70GS9gUWA98BNqB1JoQzgT/E6xHx557A07lILJcrKXJ/zDW3WNIcYCOCAyXWbZJ0GLAbcBBwuaT+ZjY0VvmRmd1VZLyY2TBgGMDbAw7K7p9NjuPUHVlOv1Mp7eXophLyvC1D0tpAT8J0Yn9aO0IjOMKbzeynBWwuteWTxE0Uf7YRZlZIsLkYn+WqFhhTjsWJ84J9x7GNAcZIehS4Cde7dBwn42R5kUmltNfU5eNAN0nfhGULPf4XGG5mOWmEQyR9TlJX4GhCpoPHgeMkbRjbfU7SlinHUiwTQj6jgQMkbZXru9IOJG0qaddEkWcvcBxnlaAeFqO0S0RnZibpGOBaSb8gONwHgJ8lqj0L/IXwzu42MxsHIOnnwCOSOgBLCdOMK+w0zGyRpFwmhE7AWOC6AvXelzQEuCf2PYfi06P5dAYuk7QpsIiQfPX00k0cx3HaH5+6TIGZvQPkL8pIMic5xZhoN4Ll782S5d0T53cBrd55mdlwYP28zQAAIABJREFUCiQ7LZEJoVfe9YPAg3llQ/Ou+xaw8xZhpWkr8rM1OI7jZInmDC8yqRRXRnEcx3GKsurHc1Q3/+rHqnEAQ9qzfT3ZyMIYsmIjC2PIio0sjKFWNlaHo70zjDttw5B2bl9PNrIwhqzYyMIYsmIjC2OolY26xx2d4ziOU9e4o3Mcx3HqGnd09cmwdm5fTzayMIas2MjCGLJiIwtjqJWNukfxhabjOI7j1CUe0TmO4zh1jTs6x3Ecp65xR+c4juPUNa6MshpTTpjaWqYiqsTeVwi5BhsSNi6sov0+wAQz+0zS14FdgSssSKiVa/vfZnZPPF/XzD6sZuwJO72BS4DtafkcW1fQ9hEzOzSe/9TMLlmRMcT22wJ/BDYys76SdgKONLNfV2FjH0KGjC0J/6+LIDVb8lnyBMhbYWYvVzGGDYCf0PrzLCiJ11bjaAskXVDJ73ctfjclnWVmV8fzHcxsavUjXo1p7x3rftTmIHyZHRzPuwJrVdBmJjAj/sw/ZlTZ/3XALcA7wC+BycCfq7QxifBlvHM8/x7wVIVtXy50vgKf47OEnIGT4mc6FPhVhW3H12IMsf1ThES+SZtTqrTxKvAlQob79XJHBe2ejMdognD6OEKOx6XAs1WO4RHgNOAV4ADgRuDSCtvWbBzR3jZAl3g+EDgHWCfFv9HbK+t3s1a/36vr4VOXdYCkbxNErP8UizYHRpZrZ2ZbmdnW8Wf+UTaCyWNvM/sm8KGZ/QrYi5BfsBoaLfyffBQhkrsCWKvCtipyXi1dLYh8y8zesiDaXTb6iNRyCXM3MxuTV9ZYpY2PzexBM5tjZh/kjnKNzOxAMzuQkBVkVzMbYGb9CcLnb1Q5hvXM7M+EfJFPmdmphCTGZanxOADuBppiSq4/A1sBt5VqIOmTIsenwKYV9lur381a2lit8KnL+uBMwl//LwKY2eu5nH2VImldoDctp5eersLEwvhzQUxH9AHhi6QaPpX0U+DrwP4xT2HnCtt2lbQL4b1zQzxf9oVglU9zLYppmF6XdBYwmxARVcLWku6L/ebOl2FmR1ZoB2CupG2IzlPSccB7lTRMTPk9Ken3wD0kkgNX8VlsZ2aTE+2mVJCoOJ+l8ed7cWr7XcIfYtVQi3EANJtZY0wR9gez/2/vvMMlLao0/ntnFIY0IAsqIjmqSEYQXERABRUVFAmKqwRBiQbYNYAsqCuIacEAooCCCIgEFQFlYMiZYYYhSJAVWUFxSYqk4d0/TjW3b8+d21XdPX1T/Z6nn9tfd1d9p8P9qurUOe/xsZJuadPmMWAj2w+3PiHpgczz9uK3uUSyexIwVdIOzU86uUYrQ1MHuvHBM7afleJ/J9XVy15dSNqTcBO+GphBzLivIX8lA/ArSUsAXwNuTuc/saA9wE7ArsAeth+StHzqL4eHgG8McZ9kS+57OQhYmHBrHQm8Bfi3zLbvabp/TGabebEvkQy8pqQHCXfyhzLbfr3leMOm+yWfxR2STgROTe0+RLggS/iSpMWBTwPHAlOBTxb2cWcP7AB4TtIuxPfZKBHWbiL1Y8KFPddAR5vVYBO9+G1OBxoTpcsZXOLMxGSmMg9qwvg4QNLRxMzzw8D+wCeA221/PrP9LGAj4Frb60pak9iX2qlDexYEpth+vKDNZOAi21t3cs5eI2kR2//oso+XAmsBD9r+S6d2AJNsP9lB25Vt39fusWHaTwE+DmyeHroc+J7tp0tt6YZe2SHptUTB42tsny5pJWAn21/tqcGVUUcd6MYBydW2B/A2wiVyEXCiM79cSTfY3kjSDGBj289ImmE72z0kaWFi1r687b1S9OIatn9V0Mf5wG4lA2RT242AB2w/lI4/DLyP2N853JkRpJLeSOzfLGp7eUnrAHvb/kRG2+8Dx9qenVYx1wBzgCWBz9g+PaOPTw33vO1vDPd8S183216/5bGb0j5Xbh8LAGsQq4a7bD/Xpklr+66iR3s9AZK0EPEbvasX/WWes+vfpqTtgJlOEciSDmvq40Dbf5hf9o8HajDK+GAh4Ee2d7T9fiKybaGC9n9Kbsdzgd9KOo/YSynhJGIf6I2NPoHsUPjE08AsST+U9N+NW2bb44FnASRtDnyVcDs9Tpke4LeAtxN7jNi+lYGVRDv+1QNh3x8Ffm/79cAGwCGZfSzW5tYWSWtKeh+wuKQdmm4foWkPNqOfLYC7geOA7wK/T59tCT8APkvaq7M9E9g5t7HtOcS+7+KF552LNFjMAC5Mx+u27qPOJ3rx2/wy8NfUx7sI9+3uwPlExHNlGOoe3fjgEmBr4O/peCEirHvTnMa2t093D5d0KbA46WJQwCq2d0p7INj+pxqbhvn8Ot06YXLTzHgn4ATbZwNnp5VqNrYfaDF9TmbTZ5vuvxU4K/X3UO5HkSJWu2UN4F3AEgzey3kS2Kugn68Db2usftLq7HRi4M5lYdvXt7z/0ujRxgTot8CL7mTbBxT2czgRtHVZaj8juS/nN734bdr2U+n+DkTqzk3ATZLaehsmOnWgGx9Msd0Y5LD99+RKzCIFfTRouEBeCfyxwIZnk1uoESW4Ck2RfjnYPqXk9S1MlvQS288TeXDNBSlLfucPSNoUcHLbHUB+4MNjabb9ILAZ4U5uBAeVrLAb+1J7MHcC/u7t2to+DzhP0httX1Ny3hZe2uzis/37tO9YQsfRo010MwFq5nnbj7cMusO699UbIYJe/DYlaVHgqdTHd5uey16lT1TqQDc++Iek9RthypI2YCDcP4dfE//wIv5pVgLuIi6yuXyRWAUuJ+k04kL/kYL2XamSECuN6ZIeId77FanPVQkXUS77AN8GliXcrxcTEZA57A38NzFJOKixJ0NcmEov1D8hEr7fDhwBfJDySMNbJO1LB4Nl4kZJP0y2kGy4qdCGoaJHP1jSQZcToGZuk7QrMfCsRkxirm7T5gsMRDReQqj1lNKL3+a3CLfrE8Adtm9MfaxH+cRh4uFRkLVeb93diIjJe4l/oCuIZNoNuuhvfeD4gtdPAj5AKG+8k3CbLdXBeTtWJUntNwG2BxZpemx1Itk4p/1k4NSR/j6TLbekvzPT35cC0wr7OItIkbiXCKm/mEjEz22/IPAp4kJ/DpEWsGBB+8nA19L9RchQ62lp/x5g36bj6wgln/uA93fwmS5M7HXdkG5fIrwhbb+H1vsdnLur32Z6/bJEsvykpseWIYJrRvw3O5pvNepynJBcSmsQq7I7XRgdN0R/c0XstXn95bZLAxVa+7jJ9gaSZjmCOJB0he1/7abfQhsuAraz/WzbF89fO663/QZJlxPpIg8B17tAsUbSLbbXkzTT9trpN3KRM3Qme4WkaZ2eT9JVwM62H0jHM4iJ0CLASba36p2l87ThTmAXYjJ3KpHn2YkQQWUEqa7LMYykLW1PU4tKArCaJJypltAS0j6JWNH9tdCc30r6DHAGgwMGSoShu1El6RX3A1elaLzm95Ed1t8jTlCo1RxKRNYtmu6X0JjsPCZpLWKwXDG3cdpvPJK5RaGnFthwS/osz2Lw55nz21ygMcglrnRImP1NkV9YRApm2dH2Y+n4ZcDPbL99mGa9EiKojCB1oBvbvBmYxuDIugYlagnNYevPE/tJZxfa0tj3ad7PMlCimdmNKkmv+N90m8TA51KiMjOJcKud2Y0RthuqMtMp+wyb6Xaw/BYR4TfLnbt+liRSNZoHhNzf5suaD2zv13S4dAe2LNUY5FJ/j6qNVJ7tLTo4T2W0MdK+03rr7kbaH+tBP1OBqT22bYHM100Blh7i8VfQZg9liDZzKeMP9VhBf1OIVUBJm8u7ON92wApNx4cBtxID1Urz87c0hC2X0rQf1MN+N8p83WnAXkM8vjdwegfnvYmm/SxipTpsJQBi//uVTccfBs4jgo6WLDz/tkM8tk8H72MyISi9fOPWz9/FWLzVPbpxQDf7Y5IOAg4mLugCHgEOs/0zSct5sOsopz8RK7Fdib2uV2S0OQG40C3uLEkfBN5k++MF5x9KDWSm7bUL+phMqMzskv5e6UjEz21/KBFdV+zGlTQT2MT2U8l1+I1kx3rEgDucm625nzcTlSRmSvoAkfR+DyGdlZX2kRQ9jiRWlc2i0MVuXIX81s7Ee3nc9oZtmpBWW+emczf2wjYggmTe6yGEltv0tw0RATo9PbQ58DHbFw3T5mai/NX/pWTvnxEye+sCryn8XVwNfMH2tHT878AWtrct6GN/IsL5YeCF9LBLft8TkTrQjQM6vbBKOpxIoN3PSf9Q0spEeP2VxGx61UwbNiYGt+0Jd9W+wPnOyDuSdLvt187judm226Y5SPo4EbSxMhFl2GAx4GrbbUPa04VsVyJy9HoiRWJlDyTqZiFpKDkmO694662210n3f0TIbh2VjrMChCR9B1ibmLzcRbgsLyQEBCbnfBapn4sJEYJZDFxUcWZSu6QViIFtF8IlvgKwoe37c9o39bMlA6kusxsDRSdIWoqIgBSheflIm9c3fx/fAf7qKN2EymXylgJ+RUwstwHWJIJtsgPHJN1DyPS1LbdUaWKkl5T11v2NDgunEvJOc7kGieTmvxOahO36+HLq5xJgTyLF4A+F9t/RyXMtr1ucCLQ4nbigNm5Z7iUiZ+5qYDdSGHzp++jRdzmTGJgmETqGGzY9d3tmH7env1OI/bHJ6VjEfluuLTd28T6uBmYTe4KrdfN5EhOvTXv0+b6MmNxt3ri1ef1twEvS/TubX09hIdzU5uXpOz6JtNAobH9pw556y7/VYJRxgO1OZYxe8BAK8A75rgdt5+gAfoxYNXwP+JXtpyWVugn+IukNbik0mlxnWdGfDiHox4FdUgDGckSw1b9IWtHtw8DPBt5LSDTNUeh9Fr2PIaJfW23MCcDoRWLw0+l8T0v6H4deJLYtqSTt5HeS3mb74oI2Df5KlH16BRE4cjedF6a9GfiCQoLsHOCMxudSgjorR9V1sreiSGtDkMHAAoTn4f2S7Iwo1qbI6PuAyyT9mi7dyROJ6rocwyR34QnAKoR7aXfb2eoZki4BvuKoqN38+JbEXkLb0OmW/awtiRnn1sByDsmjHDveAJwJnMyA8saGxMb/zravy3pD0dcRhKDyvQxcWJ35Xhr7i7sA7yACdPYALnCTxNow7U8a5mk7U5FE0rLEzP9W2y+kx5YhJLnayrJJ+hOxtyciybtxERSh2JJV+T1doBchNDyfpTC9QCHE/D7i81yV0N58e+uEJhdJS6b+diYCMFYrbN9ROSpJmxCJ2Rc7lW5Kg+6iGROoniDpi8M9795opI5b6kA3hpF0I6EMfzlRlHFPZwYrpPavIyLIriQGGBMXgs0It+XthfZMIVRRdgHeBFxie9fMti8n9vXWSg/NBo5zYR03SXcBr3eXCd8puXobUkCK7aW66a+fjMaLYvp+dyI+z+VyB9uWPt6Q+ngv4Z4dKq1muPZdl6PqBkWF8GnJ+4CiYsgWts/tx/knMnWgG8O0BifkBiu09DGFCMB4HTFjnw2cNpRLs7DfqcD27p1OYe55zwY+XjpAtulzIdtttUMlnWz7I+n+v/X7vfeatML9IJHWcKSk5YBlOl2RNfW7glNdtczXH0Xk891LrPx/4aZ8uIJ+ziFW+wcR3odHiVXyO0r76oShBlUl9ZqCPjpJep/w1IFuDCPpPuAzTQ8d03ycuSc0rpC0IbFKvY3Bexjv7sO5X7xodTLpGG1I+h4Rbbml7deki+rFtjfqsx37AD93mwjJwj7fTCpH1e3qv+Ccc6W5qEnuLrOPrgfLiUgNRhnbTGewKkrzcYkyynjiFOAoWkLi+0RPZ42S3kRELJ4kaWliT6iflaQ3tr2+pFvgRSWRBfp4/gbHA9unz8NEXuM5uY3T3l4rs9LfRYESmbpuuFHSN4DvEO9jf8qrQcyRtHxjrzalcNTVShvqQDeGsf3RkbZhfiBpkcamfwc8Yju3KnmvbXi1oiK6mu6/iAsKhaZ9tg0Joe6TiOoFpxL7p/3iuRRs5GTT0hRMHlLbA2x/s0s7vkMEs5yejveWtLXt3PJJjf3noarfZsnUpYjao4ggIVEYmJPYn0i3OCMdX0yUASrhc8CVkgYlvRf2MeGorstxgKRXAF8BXmV7W4UKxRtt/7CPNtxK/AOfYfvedq+fRx+bAicSK5flJa0D7G07u4JymjE/Q0hmNbsus6LjurFB0rC6nCV7dilgYj1CoqrhDs1SeNFgke6h7MgKRVco0+xEiHyfArwfONQFOp6SLnOXepGSZgNrOV2sFHqis5whJNArUqL2diVRzcP0tWhOFO8Q7SYR38E0CpLeK3VFN144mZj1fz4d/54YdIYd6FK49VAzncZstURW6N3ERfFMSS+k85+ZEw7fxDeJQqPnEwbcqlArKaGxV7FJ02MlKvMd29Dj4JNnU95b4+Jeota/WPuXtMf2aZJuIkrjiIh2LPk+ISpBHMfcqj0lYfl3EZqOjQCW5Yik62LSyqzhAr2iIOLx4W4HueZJFFA8kbP9gqT90kTjV93YMtGoA934YCnbZ0r6LIDt5yXNyWj3rl4ZkKLojgaOVlRvPpRw9Uwu7OeBCPZ7kZz30dz+LSWvnx829IgzJR0PLCFpL6I6xA9yGvYifSDl8i1DFH69M6UHHERUjX9VQVebpr9HNJtIxsRD0i/TaxcH7pB0fTremPaVwYfq77sMdoHuI+mtmS7QGyWdwYD2JlAc8NWLiVwvymFNOOpANz74h6R/YWAvZRMyVBtKQrxzkLQiUWl8J2JwOKSwiwfSrNcp6OEAoGgWLelAYnX7JDEwrA/8h/PVPbq2oVtSWP8ZhBbiE8Q+3WG2f5vZftg9ynZ7hQqh788TItALSvo2kXT+Y0JUOZsuJx7HDNd1B/29mcEu0FMYCEppx1TgKUIcodmGooCvHkyielEOa8JRB7rxwaeIWeIqiqrMSxO+/GHRgDTRXE9RuNEu6ToiYOIsIs/nvty2TexD6BouS2hPXszgf+gcdrf9bUlvJwIHPkoMfLkDXdc2SNrM9lXtHpsXyWV5ru0NgKzBrYXSSL5WPgas4VDsX54Y8Da3fW1pR5IWJNRMVqTpemP7iHm1aXrN9KEel7QZkft5eaE5HbtAexT41fUkyp3L/U1oajDKOEHSS4iZvwjF+xJNw16cf03bd/bznPOwY6bttdMq5DLb5/Q7z2ioHLrSvDqFUv7Jtm/ouYHtz90qRHCb7bWGazNMXxcS3oWbaFq92P56YT/rEoPbBwjR8rNtH1fYx3RC+aeR8L4RoXX5VLJprlxLSYfYPlrSsQwxKSyMpF2KmERtTfyfXgwc6IJKBArFno8T0ZYAlwHH9/v/faxRV3RjGM1bRHh1ScUJ42kfZkrjOCeQRNKHbJ8KvEPSXAoTuRF+qa+hXG6PEyr652V2c5OivMxKwGclLUZZSHzHNkh6I7EntXRL5ONUCvcqCc3NfSTdT+zFFAcISbqUoS/O7fbHWlMjXt58XHJxB15te5uC17+IQk+yUcPub4Q7V124Qw/roE1DBq9YRLqVFB2ZVSJpGL5HeE6+m453S4/t2WW/45o60I1thtP6y94/kPRu4OtEkMFfiPI2dzBQA2w4GtGAQ0X6lboLphD7Umel4/cRkmR7SHqL7YMy+tiDKIp5n6N46b8Q7st+2LAAEVH3EgZ/Hk+Q4UpuIbsY5zA0q+ZMId5LjtD2wS3H3bhCr5b0etu5e2HN3ElUC9jO9j0Akj7ZqSG2p6cE69Vs/07SQkTJmyeHafZ+oirHKepC1k3SW4g8ujXSQ3cQWq6XFXa1kVN9vMS0lNpTGQ6PglpB9TayN+BWoo7cLen4LcAJPej3oMLXT6Op1hYxYEwjVkNta7Gl1zfc8csRF6n1+mlDarNC0/1JwNQOP791gP3SbZ0efdfT+/zbup2ofHAXsR82i4jkzGm7PbGKe4AILNqKLmoEAnsBNwD3puPVCOHx4drcPNT9wvO+k3C3fjR9p+sSQSX3Ae8o7OtmYJWm45U7tWsi3eqKbpwg6Z3ECqzZ9dh2wz/xnO2/SZokaZLtSxVCut3yKaK+Wi7LEivERsToIkQS/BxJz8y7GaQQ/KOAv0s6kliV3AysJ+lHTlW656cNTfyXQp9xDrEaWlzSN2x/LbN9I3p0LwZW5adKOsH2sQV9NEtfTSIiJl+Z275HdLwydch8nZNyCN9LlBx6hUKD8xyX18nblyi6el3q/+7krp/fHAy813bzymuGovrIscAFhX1dqtC5FeF9GZcKSb2kDnTjAEnfBxYmVmInEiuZEoX5xyQtSkSxnSbpL+S5uNqaVvj6o4kLwGWp7ebAV9KF7ndt2h5E1OVbjHALrWD7EUkLE7P43IGuGxsavNb2EwplkQuAfycGvOyBjnDBbuyB+mdHEYET2QMdg12OzxOrij0K2neMojDoT4Fz3bmcGwCp/WnEb3NJYEfgP8iPpG3wjO1nG+H9KYCrnXu9F7Jur2wZ5BptZypUjbJQSLA9TgzWDSmyO23nTsAmLDXqchzQFGnY+LsoUcrkbW0b86Lqxj+JWf8HiQTd01wQDTaPfv9oe/nCNssQ/8gCrrf9v5ntmisH3OqmfYzSqMtObWhqP5twT/2U2IeZ3mpTRh+ziP2Yp9PxFOAGZyjdq0n0d6SQ9B4ikGQrohjv6UQB275UCpiHTUcDjxEFffcHPkG4oz8/TJuuZd0k3eRIFSl6ruV1exIyf/cSgVYfs31+u3aVoK7oxgeNWmlPSXoVocaelW+jEN49z/bWRHRi0WZ7m1y8hUr6SjwN/Jlwwa4qaVXbOflSC0lajxisF0j3G+K7U4Zt2TsbGhwP3E/sfV6eAiCeKLThJOA6RQ01CNddrnbpuUSiPJLOtv2+wnOT2q5ORPS9wvZaktYmCvJ+qV1bR4TqeSng493AvwHfl3QBcLozk997zL8T0YmzgL2J1faJwzVoDGSSdrR9VvNzknbMPO8qkoYalER+ovdBwOts/1XSysQKtw50mdQV3ThA0qGES2tLQukd4ETbh2a2Px/Yzany8UiRZq0HAq8GZhB6lde4fTh8I5R+njgzJL0bG4bpU8Bk223dwZJWcirFI2l9QpdRwOW2b8k8X/PqtuMcwpR3djCRp9Xor5ucurWJidTatkvTLbpCIYg8swvbO86NVNS+myeeR2L8cOfKPXclqCu6MYykjYAHbB+ZjhclZqt3Erp6uTwNzFJUL27WzyvJl+oFBxJJvNfafoukNYEs3cbcgWx+2jAvbFvSbsQqrR0/BzaQdIntrYiAmuJTzuN+KQvbvl6DJauK9m7THtQHCDfmMkTaRlHwRMO17hA1Xp1I//iNC5KkU9tbS926krYF3gEs27I/N5XMzyJnIMugdX9w0PEI/K+OKepAN7Y5nlBZQCEO+1Vi72Fd4ATyc7d+nW4jzdO2n5aEpAUdYsJrtG82Jmz4T/IGukmKWnSra4hyO85LwF9H0hMk93G6D+XSbo9IWoUBDdX3Ey7dtqQo2F2IvLFfAIc4UwJtCC4H/lVR4fwSInl7J8qTr5cBZivEoZsndMNVn//fdL53Mzi450kiCrRf9DK3ccJRB7qxzWQPqJbvROS+nQ2crahnloUjGXYhYHnbd80PQzP5k6QliD2m30p6lLjQjAkbJM1LN1FAbnTdzsR+XGvSeTY9dAvuS0yY1pT0IBG1+aHMtpsSE6/f2e620rscyf97AMc6JLmy3LgtFK/MU7TkrZJ+WrKC7DU5QS+VeVP36MYwkm4D1nWU5bmTiMS6vPFc7n6EpO0IpfgFbK+k0BU8os1Mt7n9ZOCiFNDSE9K+xuLAhSMVqVdqg6SHiTIsj7Y+BVxtO7u8jaRtbf+mxN75RXIdTvLwCiLz8/y3EBGS3wT2sD1b0qycCNQe2vAu4Egib+0llK+OKyNIXdGNbU4Hpkt6hIi8vAJA0qpklOlp4nAinP4yANszJGWrpKdk6qckLd5pQEtrsEDpvkYK3BjOxrZ7Xd3aQBTDXNT2XKtpRV5eCTdL+iEjWzX+K8DRth9Lxy8DPm37C/2yIXEQ8FkiSXx2ijocNvioGUlX2n7TEBHCJYPVt4AdiMrmRasDDdTVG5LcCWWlc+qKboyjqD23DHCxB5KLVycuuFmBDJKus71xS7TeTJcJCJ9JRCh2HNAi6TTgs53kgLWJunRu1GQ3NvQSSb8hVY23vY4iufmWPq9i5orYHIvRfpJWcJe1F9Pva6tO3LBNUZc7EMo0p6bjXYD7bX+uG9sq7akrujGOh6gRZvv3hd3cJmlXYLKiOvgBlFdw7kVASyfBAo3X9CrqsmMbekynVeN7yeQUkPMMQNrHXbCkgxTM8ifbz0jaAlgb+HFjlZjZx4bA55i7pl3uROwcus8rPAS4IKVcNFcYbxsc1PAMSDrSdnNF8V9KKqqpp1BH2Yu5P4vd59WmUge6SrA/UVH6GULN4yKgbVJwMz3aLO8qjL+BpLWA1zJY9/PH/bShB3RUNb7HnApcIumkZMfuFAoKAGcDGyZ3+g+JJOefEiH7uZxGRB3OoqDkUhPN+RGdVuL+MvB34je1QId9LC1pZaeixGl7YOnCPs4jtih+R3l18glLdV1WkLRebjLyEG3fQ9Qc+046vo6Bf95DbP+8R2bm2vNFYAtioLuAEBW+0nZpmZwRJe05HgusBdxGqhpvO6sidg/t2JaQ8RLhHr+osP3NtteXdDCRunHsUC7RNn1caftNZZbPbUPr/cI+brS9Yac2pD62IaJY70sPrUgEkGVrdkqaYXvdbuyYiNSBrtLYf2gk8/7M9uyCtlcBO9t+IB3PIC6MiwAnpaTn3L42IS7uryFmzZOBf5REtik0Itch9rPWUSQsn2h7uNp9PbWhV2iEq8b3gjTx+RbhMdjO9h9KIoJTH1sR+1mXMNhtmFtvcQ4DxWsXIlUUpyAYRdJXgWklg9I8+lmQSHiHDgSZJX0hll5NAAALjElEQVSJiOAtqXgw4amuywoOBZBXEgoWJ0iaCpzhDE1DIiXhgabjKx1i0H9LYeklHEfkkZ0FbEiI765W2EdDQeP59D7+Qpm7qhc2dIVCG/MfjuoLixEyYKsSuX39tGMHoupDQym/k5D6jwL7AF9Og9xKDARjlPSxJlFZu+G6zC4s3KO8wn2BQxSlmp6jg89C0ksJjc3GPt1lko4vnMAcCHxO0rNEnb+a5pBBXdFVBiHp9cTG+0622+5FSLrH9qrzeO5e26sUnPtG2xs2R3xKutr2pgV9fJcIXNgZ+DSxrzLDdpbsVC9s6AaFbulHiAv5zwjlm8uAjYFbnVdlvVe23EOswu7o1znnYUdfc+bmF5JOJAbrxj7nbsAc23uOnFUTg7qiqyDpNYSyyo7AI8QF9tOZza+TtJftH7T0uTdlNfEgqi8sQNSDO5qQmypaFdr+RLr7fUkXEtW9S/a1urahS3Yh3KYLA38kapk9ldyY2Wo3PeLhbge5FMX7X8wdHFSyyr5W0mtt396NLd0gaTNiwvQPSR8ioji/VZiGspEHl2qaJmmuOnVt7BAhfbaS7SMlLQcsY7v0f21CUVd0FSRdSySfn+Xy2msvJ1xqzzAgQLwBEYb+XtsPF/S1AvAwsTf2SUKV5Lu27ynooyGGPOxj89OGbmgJnBgUtNHvHDZJ3ybyvhrfL5C/N5b6uBL4IqFqsh3hhpTtLxb0cQdRVPcPyY6Guy47z7NbFPJu6xDpET8hIkh3sD1sZYKWPm4GdrR9bzpeGfh5yXeqqK7+ArCl7dcokvgvtr1R/ruZeNQVXQWiMvkqwMsk/Z9Tsc8cbP8F2FTSlsDr0sO/tj2t1IimpN6nKQzzVxQmXRhYKv3zN0LKpwLZ0lvd2NAjlkh7YwKmpvuk48X7bMtUInCjuYBv9t5YYiHbl0hS+mwPl3QFMfjlsk3Ba+cXz9t2ijL+tu0fqk1R1iE4GLhU0n3E97kChZUciKrz6ytpfdp+NHkgKsNQB7oJTHKHfYX4Z/sjUbT01Slv6vMlm+RpYCse3JIdvUhR2JuQinoVg0vbPMFAjb75bUMvmE6sfCBU+5ujRYuSi7sld1+zDU8rpNXulrQf8CAR3FJixyBVE4Xo9r5Eblu/eFKRvP8hYHOFvutLSzpIA/5qDETSFkddAs+lczfyK5ems9zCCUV1XU5gJH2TUMj/pJNgb4pUPIaIXjywT3b0MkVhf9vHjqQNYx1JhzgqBBzLEBqNLpN12wi4A1iCEEVenNDPnEvRZ4i2ywGHEpOXc4lE8yOJII7T+/X7TLa8EtgVuMH2FZKWB7ZwvhBBI+ry4zRFXRJFbbMnlJI+SOynbwCcTJTi+oJbqp9XBlMHugmMpLuB1d3yI0gzxjtt9yWsXtINzXsMko6zvV+6f63tTQr6WoAIZy+6mPTShrGOpO1s/3Jerjn3qWSMIr9zOnAN4b7cCphNTMwe6ocN87BrKeBvrf83Ge16EnWpKAbcmHhNG+mo2LFAHegmMJJ+b3v10ufmgx29TFHo6GLSSxvGI2nf87Hci7uk84d73hnaoZJubY5SVJRBWr4Dd1/HKAQEvgr8H7Ga/AmwFOHm/7DtCwv6GvR+5vVYRj/rE7mVBq5ypnj7RKbu0U1sbpf04Vb3SwqfvrOPdnSdoiDpJbafp/MQ7l6mSYxpJB0GnOmorr4g8Buiav3zkna1/buMbt4IPEBE814Hg/QmS2xpDix6CFhYSYjAA0WH5yfHEXmZixN70Nvavjatqk4Hsgc6YI6kVVqiLov0KtN3syOhISrgJElnOU/cYcJSV3QTGEnLEhF0/wRuImaIGxEySdvbfrBPdnSdoqABTcWOQrh7mSbRCyTtSBR8fVLSF4i8rS/1Y/YuaTawVooy/BiR27c1sDpwiu03ZPQxGXhrars2UdnidJfJy91PBFoMNUi6MBevI9SkLSnpDtuvaXquVLNzK6L00qCoS9sltfXuANZrREYrKkrc3GxXZW7qim4CkwayjZtSAwT8xvYlfbajFykKjYvhZxgI4YYQzm0bPdjLNIkecajtsyS9iahafgzwPUIhZX7zbJOL8u2E/ukc4I4UqduW9PoLgQvTqnAXQvLqiNxgIdsrlpvec5ojGv/Z8lyuG/cg4Cpiv7HbqMv7icT7RgrQgsC9hX1MOOqKrjIukPQnoFEbbCGSGDNxUfinM+qGjSYaqwVJ/0VUtf5p6Qqii3NfC+xJJM7fBWxg+w/puTttrzlc+6Z+FgTeSQxyKxIlen5U6ilQlyIA3aDhBaGn2G6bYiDpGGBTQq9zJlHr8SrgmlL3q6RzCa/Lb4mB9q3AlYSma1FE7ESirugq44XJwKIMdnMtmv4u1n9zuuZBSccTLsOj0qAxqU/nPhD4OZFH+M2mQe4dQFY5J0mnECWGfgP8p+3bSo1QiAAsQpciAN3gHghC2/4MvBgRvCEx6O0O/EDSY7ZfW9DdOenW4LJu7ZsI1BVdZVygPstjzW8kLUyE1M+yfbekZYDXu8syMf1C0gsMVGhvvsiUlMY5kAERgGZpuieAH9g+rkfm9gVJixNBOpulv0sQ3292Yn4a/FclPtN7XaBiNJGpA11lXNAvt14/SEoiM11Qs20806kIwGhB0gnEvu+TRATqtcC1th8t6KOhYrQ78D8kFSMiuKVIxWgi0i9XSKUyvxk3yiW2XwBuTeobFThe0gGSfp5u+yWVkbHC8kTQyEOEBNqfgMcK+/gasCRRtWCDNKlbhVgVHtNDW8cldUVXqYxCJE0jgg6uZ8AFmJVoPd7olaLISCJJxKpu03Rbi0hCv8YZlRxGi4rRWKUGo1Qqo5ORqJwwiJHM5Uvn71YEYNSQBqjbJD0GPJ5u7wLeQF4lBw+lSmN7jqS6WmlDHegqlVGI7ekjbQMjm8sHsZpdnx4oiowkkg4gVnGbAc+RUguAHwGzMrsZLSpGY5I60FUqo5CksXgsUW18AVJeYE60Yg9pDCbvBL5n+zxJh/fx/F2JAIwiViTSNT5p+88d9rEv8AtJuzOEilEvjBzP1D26SmUUIulGYGfgLCL36sPAarY/10cbfkUET2xNyKH9E7i+VIS4i/OPKxGAXtCiYjS73ypGY5W6oqtURim275E0OclpnSTp6j6b8AEil+8Y24+lXL6D+3j+8SYC0DXuosDxRKYOdJXK6OSppKQxQ9LRwJ8JlZC+kHL5rm/O5Utut05db53wZ9tH9PF8lXFKzaOrVEYnuxH/n/sR7rrlgPf16+SjJJevo9I+lUordY+uUhmlpBIsy9u+a4TOP6K5fJKW7FPNuco4pw50lcooRNJ2RDj/ArZXkrQucEQ/E8YlvXmox0dJ6kOlkk0d6CqVUYikm4AtgcsaGp6SZtpee2Qtq1TGHnWPrlIZnTxv+/GRNEDSJpJukPR3Sc9KmiPpiZG0qVLphDrQVSqjCEkXSFqJkIvaFZgsaTVJxxIFO/vJcUTR1LuJPLY902OVypiiDnSVyujiZOAi4H5C+PcZ4KeENuKB/TbG9j3AZNtzbJ8EbNFvGyqVbqkDXaUyirB9JrAekRj9TuAM4GfAo4QMVD8ZlMsn6ZP0MZevUukVdaCrVEYfzxHh/AsSA17j1m81kBHN5atUekWNuqxURhGStiH0Hc8n0gmeGmF7RjSXr1LpBXVFV6mMLj4P7Gj7P0bBILcdMAO4MB2vK+n8kbSpUumEuqKrVCpDUnP5KuOFuqKrVCrzYsRz+SqVXlAHukqlMohRlstXqXRNHegqlUorJzOKcvkqlW6pe3SVSmUuJC0CHEYUXv0J0LhQeCJW9q6MbWrh1UqlMhStuXx1RlwZs9SBrlKpDKIll2/9kU5zqFS6pbouK5XKICRdAexje/ZI21Kp9II60FUqlUplXFOjLiuVSqUyrqkDXaVSqVTGNXWgq1Qqlcq4pg50lUqlUhnX1IGuUqlUKuOa/wfMQ6UXA+9WFAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot correlation heatmap to visualize results. \n",
    "\n",
    "\n",
    "corrmat = transform_df[correlated_cols].corr()\n",
    "sns.heatmap(corrmat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorical Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following features are categorical variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "nominal_features = [\"PID\", \"MS SubClass\", \"MS Zoning\", \"Street\", \"Alley\", \"Land Contour\", \"Lot Config\", \"Neighborhood\", \n",
    "                    \"Condition 1\", \"Condition 2\", \"Bldg Type\", \"House Style\", \"Roof Style\", \"Roof Matl\", \"Exterior 1st\", \n",
    "                    \"Exterior 2nd\", \"Mas Vnr Type\", \"Foundation\", \"Heating\", \"Central Air\", \"Garage Type\", \n",
    "                    \"Misc Feature\", \"Sale Type\", \"Sale Condition\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to find features which are currently numerical but need to be encoded as categorical variables as the numbers they have don't have any semantic meaning. \n",
    "\n",
    "However, if some categorical features have hundreds or thousands of unique values (or categories) then we may have to drop these features as otherwise hundreds of columns will have to be added back to the data frame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We filter for categorical variables which we still have after cleaning and filtering our data.\n",
    "\n",
    "transform_cat_cols = []\n",
    "for col in nominal_features:\n",
    "    if col in transform_df.columns:\n",
    "        transform_cat_cols.append(col)\n",
    "\n",
    "# How many unique values in each categorical column?\n",
    "uniqueness_counts = transform_df[transform_cat_cols].apply(lambda col: len(col.value_counts())).sort_values()\n",
    "\n",
    "# We use an aribtrary cutoff of 10 unique values (however this could be higher). \n",
    "drop_nonuniq_cols = uniqueness_counts[uniqueness_counts > 10].index\n",
    "transform_df = transform_df.drop(drop_nonuniq_cols, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Select just the remaining text columns and convert into categorical features. \n",
    "text_cols = transform_df.select_dtypes(include=['object'])\n",
    "for col in text_cols:\n",
    "    transform_df[col] = transform_df[col].astype('category')\n",
    "    \n",
    "## Create dummy columns and add back to the dataframe!\n",
    "transform_df = pd.concat([transform_df,pd.get_dummies(transform_df.select_dtypes(include=['category']))], axis=1).drop(text_cols,axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can incoporate this cleaning of categorical features into the select_features() function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def select_features(df, coeff_threshold=0.4, uniq_threshold=10):\n",
    "    abs_corr_coeffs = abs(transform_df.corr()['SalePrice']).sort_values()\n",
    "    df = df.drop(abs_corr_coeffs[abs_corr_coeffs < coeff_threshold].index, axis=1)\n",
    "    \n",
    "    nominal_features = [\"PID\", \"MS SubClass\", \"MS Zoning\", \"Street\", \"Alley\", \"Land Contour\", \"Lot Config\", \"Neighborhood\", \n",
    "                    \"Condition 1\", \"Condition 2\", \"Bldg Type\", \"House Style\", \"Roof Style\", \"Roof Matl\", \"Exterior 1st\", \n",
    "                    \"Exterior 2nd\", \"Mas Vnr Type\", \"Foundation\", \"Heating\", \"Central Air\", \"Garage Type\", \n",
    "                    \"Misc Feature\", \"Sale Type\", \"Sale Condition\"]\n",
    "    \n",
    "    transform_cat_cols = []\n",
    "    for col in nominal_features:\n",
    "        if col in df.columns:\n",
    "            transform_cat_cols.append(col)\n",
    "\n",
    "    uniqueness_counts = df[transform_cat_cols].apply(lambda col: len(col.value_counts())).sort_values()\n",
    "    drop_nonuniq_cols = uniqueness_counts[uniqueness_counts > 10].index\n",
    "    df = df.drop(drop_nonuniq_cols, axis=1)\n",
    "    \n",
    "    text_cols = df.select_dtypes(include=['object'])\n",
    "    for col in text_cols:\n",
    "        df[col] = df[col].astype('category')\n",
    "    df = pd.concat([df, pd.get_dummies(df.select_dtypes(include=['category']))], axis=1).drop(text_cols,axis=1)\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choose the Type of Cross Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We add a parameter, k, to control the type of cross validation that occurs to validate the regression model. \n",
    "\n",
    "The k paramter accepts integer values with a default value of 0. When k=0, holdout validation is implemented (this splits the data into 2 parts to test and train) which is what is already currently implemented in the train_and_test() function. \n",
    "\n",
    "When k=1, simple cross validation is performed. The ordering of rows is shuffled and the first half assigned to fold_one and second half to fold_two. We train on fold_one and test on fold_two, then train on fold_two and test on fold_one and then return an average RMSE value for each validation. \n",
    "\n",
    "When k>0, this is also known as k-fold cross-validation where the average RMSE value is returned over all (k-fold) train-test splits. \n",
    "\n",
    "We subsequently update the train_and_test() function to incorporate this parameter k which returns the average RMSE value and standard deviation of RMSE values. The other final functions for transform_features() and select_features() are also attached below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def transform_features(df):\n",
    "    num_missing = df.isnull().sum()\n",
    "    drop_missing_cols = num_missing[(num_missing > len(df)/20)].sort_values()\n",
    "    df = df.drop(drop_missing_cols.index, axis=1)\n",
    "    \n",
    "    text_mv_counts = df.select_dtypes(include=['object']).isnull().sum().sort_values(ascending=False)\n",
    "    drop_missing_cols_2 = text_mv_counts[text_mv_counts > 0]\n",
    "    df = df.drop(drop_missing_cols_2.index, axis=1)\n",
    "    \n",
    "    num_missing = df.select_dtypes(include=['int', 'float']).isnull().sum()\n",
    "    fixable_numeric_cols = num_missing[(num_missing < len(df)/20) & (num_missing > 0)].sort_values()\n",
    "    replacement_values_dict = df[fixable_numeric_cols.index].mode().to_dict(orient='records')[0]\n",
    "    df = df.fillna(replacement_values_dict)\n",
    "    \n",
    "    years_sold = df['Yr Sold'] - df['Year Built']\n",
    "    years_since_remod = df['Yr Sold'] - df['Year Remod/Add']\n",
    "    df['Years Before Sale'] = years_sold\n",
    "    df['Years Since Remod'] = years_since_remod\n",
    "    df = df.drop([1702, 2180, 2181], axis=0)\n",
    "\n",
    "    df = df.drop([\"PID\", \"Order\", \"Mo Sold\", \"Sale Condition\", \"Sale Type\", \"Year Built\", \"Year Remod/Add\"], axis=1)\n",
    "    return df\n",
    "\n",
    "def select_features(df, coeff_threshold=0.4, uniq_threshold=10):\n",
    "    abs_corr_coeffs = abs(transform_df.corr()['SalePrice']).sort_values()\n",
    "    df = df.drop(abs_corr_coeffs[abs_corr_coeffs < coeff_threshold].index, axis=1)\n",
    "    \n",
    "    nominal_features = [\"PID\", \"MS SubClass\", \"MS Zoning\", \"Street\", \"Alley\", \"Land Contour\", \"Lot Config\", \"Neighborhood\", \n",
    "                    \"Condition 1\", \"Condition 2\", \"Bldg Type\", \"House Style\", \"Roof Style\", \"Roof Matl\", \"Exterior 1st\", \n",
    "                    \"Exterior 2nd\", \"Mas Vnr Type\", \"Foundation\", \"Heating\", \"Central Air\", \"Garage Type\", \n",
    "                    \"Misc Feature\", \"Sale Type\", \"Sale Condition\"]\n",
    "    \n",
    "    transform_cat_cols = []\n",
    "    for col in nominal_features:\n",
    "        if col in df.columns:\n",
    "            transform_cat_cols.append(col)\n",
    "\n",
    "    uniqueness_counts = df[transform_cat_cols].apply(lambda col: len(col.value_counts())).sort_values()\n",
    "    drop_nonuniq_cols = uniqueness_counts[uniqueness_counts > 10].index\n",
    "    df = df.drop(drop_nonuniq_cols, axis=1)\n",
    "    \n",
    "    text_cols = df.select_dtypes(include=['object'])\n",
    "    for col in text_cols:\n",
    "        df[col] = df[col].astype('category')\n",
    "    df = pd.concat([df, pd.get_dummies(df.select_dtypes(include=['category']))], axis=1).drop(text_cols,axis=1)\n",
    "    \n",
    "    return df\n",
    "\n",
    "def train_and_test(df, k=0):\n",
    "    numeric_df = df.select_dtypes(include=['integer', 'float'])\n",
    "    features = numeric_df.columns.drop(\"SalePrice\")\n",
    "    lr = linear_model.LinearRegression()\n",
    "    \n",
    "    if k == 0:\n",
    "        train = df[:1460]\n",
    "        test = df[1460:]\n",
    "\n",
    "        lr.fit(train[features], train[\"SalePrice\"])\n",
    "        predictions = lr.predict(test[features])\n",
    "        mse = mean_squared_error(test[\"SalePrice\"], predictions)\n",
    "        rmse = np.sqrt(mse)\n",
    "        std_rmse = 0\n",
    "\n",
    "        return rmse,std_rmse\n",
    "    \n",
    "    elif k == 1:\n",
    "        # Randomize *all* rows (frac=1) from `df` and return\n",
    "        shuffled_df = df.sample(frac=1, )\n",
    "        train = df[:1460]\n",
    "        test = df[1460:]\n",
    "        \n",
    "        lr.fit(train[features], train[\"SalePrice\"])\n",
    "        predictions_one = lr.predict(test[features])        \n",
    "        \n",
    "        mse_one = mean_squared_error(test[\"SalePrice\"], predictions_one)\n",
    "        rmse_one = np.sqrt(mse_one)\n",
    "        \n",
    "        lr.fit(test[features], test[\"SalePrice\"])\n",
    "        predictions_two = lr.predict(train[features])        \n",
    "       \n",
    "        mse_two = mean_squared_error(train[\"SalePrice\"], predictions_two)\n",
    "        rmse_two = np.sqrt(mse_two)\n",
    "        \n",
    "        avg_rmse = np.mean([rmse_one, rmse_two])\n",
    "        std_rmse = 0\n",
    "        \n",
    "        return avg_rmse,std_rmse\n",
    "    \n",
    "    else:\n",
    "        kf = KFold(n_splits=k, shuffle=True)\n",
    "        rmse_values = []\n",
    "        for train_index, test_index, in kf.split(df):\n",
    "            train = df.iloc[train_index]\n",
    "            test = df.iloc[test_index]\n",
    "            lr.fit(train[features], train[\"SalePrice\"])\n",
    "            predictions = lr.predict(test[features])\n",
    "            mse = mean_squared_error(test[\"SalePrice\"], predictions)\n",
    "            rmse = np.sqrt(mse)\n",
    "            rmse_values.append(rmse)\n",
    "        avg_rmse = np.mean(rmse_values)\n",
    "        std_rmse = np.std(rmse_values)\n",
    "        return avg_rmse, std_rmse \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the following average RMSE and standard deviation of RMSEs is obtained with a minimum correlation level of 0.3 and k-value of 4:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29019.31888760192\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv(\"AmesHousing.csv\")\n",
    "transform_df = transform_features(df)\n",
    "filtered_df = select_features(transform_df, 0.3)\n",
    "rmse = train_and_test(filtered_df, k=4)\n",
    "\n",
    "print(rmse[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test and train models with different k-values to find which generates the optimal results i.e. has the lowest RMSE value. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33295.52198740695, 30264.80627841288, 30174.953453571914, 29231.876352393345, 28777.799530731216, 28903.74548133644, 29163.097565365693, 28436.02859727205, 28692.99863781464, 28465.64728353052, 28298.779499339435, 28689.390265474547]\n",
      "[0.0, 0.0, 1509.3939694318742, 2791.201898558056, 4031.695481486712, 4473.491414706776, 6607.457785033523, 6141.809485281336, 6226.955664119915, 6212.713548095368, 8033.782618377063, 7860.085131282081]\n"
     ]
    }
   ],
   "source": [
    "rmses = []\n",
    "stds = []\n",
    "\n",
    "for k in range(0,12):\n",
    "    rmse = float(train_and_test(filtered_df, k)[0])\n",
    "    std = float(train_and_test(filtered_df, k)[1])\n",
    "    rmses.append(rmse)\n",
    "    stds.append(std)\n",
    "\n",
    "print(rmses)\n",
    "print(stds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1ba443fbd30>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdgAAAEGCAYAAADG7YTGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd1RU19rH8e8GBOwVY0FEYwW7iFhj1JjYa0RjEkts0djSzL0xvdx4TVETy2svSRRbLIk9GjV2bCiKih0VxQY2+n7/GLzBPsAMZ8rzWWsWzJwz5/yYGB72PvvsrbTWCCGEEMKyXIwOIIQQQjgiKbBCCCGEFUiBFUIIIaxACqwQQghhBVJghRBCCCtwM+rERYoU0b6+vkadXggh7NKePXuuaK29jM5xz549e4q6ublNA6rgXI22VOBQcnJy39q1a19+1A6GFVhfX19CQ0ONOr0QQtglpdQZozOk5+bmNq1YsWKVvby8rru4uDjNfZ+pqakqJibGLzo6ehrQ7lH7ONNfG0IIISyvipeXV5wzFVcAFxcX7eXlFYup5f7ofbIxjxBCCMfj4mzF9Z60n/uxdVQKrBBCCGEFUmCFEELYNVdX19qVKlXyK1++vH/Tpk3LXblyxRXg6NGj7kqp2sOGDStxb9+LFy+6ubm51Xr99dd9AA4cOOARGBhYsVKlSn5ly5b17969e2mA33//PW/evHlrVKpUye/eY+nSpXkzksuwQU5CCCGEJXh4eKRGREQcBujUqZPvmDFjvEaPHh0N4O3tnbB27doCwAWAOXPmFCxXrlz8vfcOHjzYZ+jQoZdeffXVGwC7du3KeW9bQEDArY0bN0ZmNtdTW7BKqRlKqctKqUOP2a6UUuOVUpFKqTClVK3MhhFCCCGyIigo6Pb58+fd7z339PTU5cqVu7t58+ZcAIsXLy7UoUOHa/e2X758OUfp0qUT7z0PDAy8a6ks5rRgZwE/AXMes70lUD7tUReYlPZVCCGEE+mzrE+pQ5cP5bLkMasUrXJnRvsZ58zZNzk5mY0bN+Z94403rqR/vVu3btd+/vnnQiVLlkxydXXVJUqUSLpw4YI7wODBgy+1atWqQs2aNW83a9YsdvDgwVeLFCmSAhAaGpqnUqVKfveOs3jx4hP+/v4J5mZ/agtWa70ZuPaEXdoDc7TJDqCAUqq4uQEy6nDMYUasHkFiSuLTdxZCiHui10PEOLiwBm6fAZ1qdCJhIQkJCS6VKlXyK1iwYI0bN264dejQIS799s6dO8dt2rQp36xZswp17tz5vno2bNiwqwcPHgzv1KnTtc2bN+etU6dOpbt37yowdRFHREQcvvfISHEFy1yDLQmk/+siKu21iw/uqJTqD/QH8PHxydTJTl0/xdidY2lWthltKrTJ1DGEEE7m9hnY1B5S7vzzmmtOyFcR8laEfJVMj/yVIG8FcLNoI8xpmNvStLR712CvXr3q2qJFi3LffPNN0VGjRv1vdiVPT09drVq1O5MmTSp26NChQwsWLCiQ/v2+vr5Jw4cPvzp8+PCr5cuX9w8NDc358FkyzhIFVj3itUfeE6W1ngJMAQgICMjUfVMvPPsCBT0LEhIeIgVWCGGe0KGmry/theSbEHcU4iJMj2u74ewC7vu1lcvn/qJ773vPYqAe9StP2ILChQunjB8//myXLl3KvffeezHpt40cOTK6cePGN4sVK5aS/vVFixbla9u27U0PDw999uxZtxs3briWLl06MSwsLMtF1hIFNgoole65N2mjtazB3dWdTpU7sSB8AfHJ8Xi6eVrrVEIIRxC1HM4vhxqjoVBN02tFG9+/T0o83DyeVnTTFd+T0yH59j/7ueV9uPDmrQh5y4GrR/b9TOKxGjRocLdy5cp3p02bVrB58+a37r0eEBAQHxAQEP/g/qtXr8737rvv+nh4eKQCfPbZZ1E+Pj7JYWFhD12DHTly5MXevXtfNzeL0vrpDUmllC/wu9b6oSmhlFKtgbeAVpgGN43XWgc+7ZgBAQE6s3MRrzuxjhY/t2BJ1yV0rNwxU8cQQjiB5Nvwhz+45YaW+8ElR8berzXcPf9w4Y2LgDtR/+ynXCB32XTdzOm6nT2LWPRHUkrt0VoHWPSgWXDgwIHT1atXv/L0PR3TgQMHilSvXt33Udue2oJVSs0DmgBFlFJRwCdADgCt9WRgJabiGgncAXpbJPUTPF/mebxyeTE/fL4UWCHE4x360nT9tfmmjBdXMHUH5/I2PYo1v39b0k24eezhwhu9DlLTjYXxKPxPsb1XeAvVNB1TOLSnFlitdfenbNfAYIslMoObixudK3dmTtgcbifeJrd77uw8vRDCHsQehiPfQpmeD3cJW0KOvFCotumRXmoK3DkDsRFwM13xPb8C4qeb9qn8HtT8r+UzCZtitzM5BVcJZvKeyfx+7HeCqwQbHUcIYUu0ht2DTEWw5pjsPbeLK+Qpa3rQ6v5tCddMLV4Py3YbC9tkt3MRN/JpRLE8xQgJDzE6ihDC1pyaC5c3QY1vwNNm1iYHj0LgVQ/ylTc6icgGdltgXV1cednvZVYeX0lcQtzT3yCEcA4J12Dfu1A4CJ7ta3Qa4cTstsACdKvSjYSUBJYfXW50FCGErTjwb0i8CoGTTKN7hTCI3V6DBQjyDqJUvlKEhIfwarVXjY4jhDDalZ0QOQUqDoOCNYxOI7LRyJEjiy1evLiwi4uLdnFxIX/+/MmxsbFud+7ccbl+/bpbyZIlEwF+/PHHMx9++KH35cuXc7i7u6cmJSWpRo0a3fz+++/P35uD2FLs+s87F+VCV/+urIlcw/W7Zt/7K4RwRKnJsHsg5CwO1T43Oo3IRuvXr8+9Zs2aAgcPHjx87Nixwxs3bjwWEhJyKiIi4vDEiRPPpJ9T+IUXXrgNMGfOnJPHjh07fOTIkcMeHh6pLVu2LGfpXHZdYAGC/YNJSk3it4jfjI4ihDDS8YlwfT/UHmsaPSycxvnz53MUKlQoOWfOnBqgePHiyb6+vknmvNfT01NPmjQp6sKFC+7bt2+3yBzE99h1FzFAQIkAyhYsS0h4CH1q9jE6jhDCCHcuwIFRUPxFKNXF6DTOa0efUtyw7HJ1FKhyh6AnLyLQoUOHuP/85z8lfH19qzRs2DCue/fu11q3bn3rSe9Jz83NjcqVK985dOiQZ7169Sy2Hqzdt2CVUnT168qfJ/8k5nbM098ghHA8e9+G1EQI+Ekm43dC+fPnTz106NDhn3766YyXl1dyz549nx0/fnzhjBzDnGmDM8ruW7BgmnTim63fsOTIEgYEDDA6jhAiO11cB2dDoOpnpkn3hXGe0tK0Jjc3N9q0aXOzTZs2N6tVq3Z37ty5hYcOHXrVnPcmJydz9OjRXNWqVbPoQjV234IFqP5MdSoWriiTTgjhbFLiTTM25S0Pfu8bnUYY5MCBAx4HDx7833JG+/bty+nt7Z1oznsTEhLUW2+95V28ePHEunXrWqx7GBykBauUItg/mC+3fEn0rWiK5SlmdCQhRHY4PBpuRcLza8FVlq50VnFxca5Dhw71iYuLc3V1ddW+vr4Js2fPPvOk97z++utl3d3dUxMTE10aNWoUt2rVqkhL5zJruTpryMpydY9yOOYw/hP9+bHlj7wV+JbFjiuEsFE3I+GPKuDdARrONzpNtpHl6mzLk5arc4guYgA/Lz+qFK3C/EPO8z+aEE5La9g9GFzcodb3RqcR4pEcpsCC6Z7Yree2ci7WsOvsQojscHYhRK+F6l9CrhJGpxHikRyuwAIsPLzQ4CRCCKtJioO9w6FgTSg/yOg0AlJTU1Od8t6otJ879XHbHarAli9cnprFaspoYiEcWdgncDca6kwGF4cYp2nvDsXExOR3tiKbmpqqYmJi8gOHHrePw/3rDPYP5oM/P+DU9VOUKVjG6DhCCEu6vh+OjYdyA6BIoNFpBJCcnNw3Ojp6WnR0dBUcrNH2FKnAoeTk5Meuiegwo4jvOX3jNGXGleGbZt8wsuFIix9fCGEQnQprG8Dtk9AmAtwLGp3IELY2ilg8nsP9teFbwJe6JetKN7EQjubENLi6A2p+67TFVdgXhyuwYOom3he9j2NXjxkdRQhhCfGXYf8HULQJ+Mraz8I+OGSBfdn/ZQBCDkkrVgiHsO99SL4FdSbKZP7CbjhkgfXO501Dn4bSTSyEI7i8GU7NhkrvQv7KRqcRwmwOWWDB1E0cHhNO+OVwo6MIITIrJRF2vwm5faHKKKPTCJEhDltgu/h1wUW5SCtWCHt29AeIPQwBP4KbZdfxFsLaHLbAFstTjCa+TQgJD7HKQrpCCCu7fQYOfg7e7aFkG6PTCJFhDltgwdRNfOzqMQ5cOmB0FCFERu0ZZvpae7yxOYTIJIcusJ0qd8JVucpoYiHsTdQKiFoGVT+B3D5GpxEiUxy6wBbJVYTmZZtLN7EQ9iT5NuwZAvn9odIIo9MIkWkOXWDB1E186sYpdl/YbXQUIYQ5Dn1puv5aZxK45DA6jRCZ5vAFtkOlDuRwySHdxELYg9jDcORbKNsLijYyOo0QWWJWgVVKvaSUOqqUilRKffCI7fmVUiuUUgeUUuFKqd6Wj5o5BXMW5MVyL7Lg8AJS9WOX7RNCGE1r2D0IcuSFGv81Oo0QWfbUAquUcgUmAC0BP6C7Usrvgd0GA4e11tWBJsB3Sil3C2fNtG7+3YiKi2L7ue1GRxFCPM6puXB5E9QYDZ5eRqcRIsvMacEGApFa65Na60RgPtD+gX00kFcppYA8wDUg2aJJs6BdxXZ4unnKpBNC2KqEa7DvXShSD559w+g0QliEOQW2JHAu3fOotNfS+wmoDFwADgLDtH64P1Yp1V8pFaqUCo2Jiclk5IzL65GXVuVbsfDwQlJSU7LtvEIIMx34NyReMw1sUg4/NEQ4CXP+JT9q6YoH73l5EdgPlABqAD8ppfI99Catp2itA7TWAV5e2dsFFOwfTPStaLac3ZKt5xVCPMWVnRA5BSoMhYLVjU4jhMWYU2CjgFLpnntjaqmm1xtYok0igVNAJctEtIzW5VuTK0cuGU3s6G4cgsQbRqcQ5kpNNk3mn7MEVPvM6DRCWJQ5BXY3UF4pVSZt4FI3YPkD+5wFmgEopZ4BKgInLRk0q3K756ZthbYsOrKI5FSbuTwsLOn6flhZDX4rCTv7m54L23Z8IlzfB7XHmkYPC+FAnlpgtdbJwFvAGuAIsEBrHa6UGqiUGpi22xdAfaXUQeBPYKTW+oq1QmdWsH8wV+5cYcOpDUZHEdZwYBS4FwDf7nD6Z1hVE9Y2gNO/QkqC0enEg+5cMP03K/4ilOpsdBohLE4ZNYVgQECADg0NzdZzxifHU3RMUV72e5np7adn67mFlcVshXUNocY34DcSEq/DyVlwbCLcigTPovBsXyg3EHKXeurhRDb4uxtELYXWhyBvOaPT2A2l1B6tdYDROcTTOdVwPU83TzpU6sCSiCUkpiQaHUdYitamUaiexaDCW6bX3Aua5rFtexSarIbCdSH8P7DcFzZ3hOj1pvcJY1xcB2dDwP/fUlyFw3KqAgumbuIb8TdYd2Kd0VGEpUSvg8ubocoocMt9/zblAiVehOeWQ7uTUPl9iPkbNrwAf1SGiHEyKCq7pcSbZmzKW97U2yCEg3K6AvvCsy9Q0LOgTDrhKLSGAx9C7tLwbL8n75vHF2r8Bzqcg3pzIEcB2DvcNChq1wC4HpYtkZ3e4dGmbvs6E8HVw+g0QliN0xVYd1d3OlbqyNKIpcQnxxsdR2RV1FK4FgpVPwVXM2fndPWEMq/BizvgpVAo3Q1OzYFV1WFdIzg9H+QSgnXcjDR11ZfuBsWaG51GCKtyugILEFwlmJuJN1kdudroKCIrUlMgbBTkqwS+r2buGIVqQ9B06HAean4Ldy/Ctu6wzAcOfAR3oiyb2ZlpDbsHm1qttb43Oo0QVueUBbZpmaYUyVWE+YfmGx1FZMWZX03Lm1X7AlzcsnYsj0JQ+R1oewyarIRCARD+FSzzhS2dIXqDDIrKqrMLIXotVPsSchY3Oo0QVueUBdbNxY3OlTuz4tgKbifeNjqOyIyURAj7BArWglKdLHdc5QIlWkKT36HdCaj0jmmFlw3N4A8/OPojJMZa7nzOIinOdL27YC0oP8joNEJkC6cssADdqnTjTtId/jj+h9FRRGacnA63T0H1r6w3OXyeMlBzNHSIgqBZkCMf7BkKS0vCrjdN0zIK84R9AnejIXAyuLganUaIbOFUE02kl5KagvcP3tQvVZ/FXRcblkNkQvIdWFEO8pSD5ptAPWo9Ciu5GgrHJ8DpeZCaAEUbm1pk3h3NH2RlCSmJptVnEq6kPa6aviZehfi0rwlXTC1H5QLKFZRb2te0h0v6526PeT2z29J9n3wLQgdBuQGmkcMiS2SiCfuRxQtX9svVxZWX/V5m6t6p3Ey4SV4PmQfVbhybYBqM1CAke4srQOEAKDzTNCDqxAw4Pgm2djNNclGuv+mR68HVHJ8iJT6tQKYrkumL5oNfE6+aCufjuOUG98LgUcTU6iYVUhMh9S7oFNDJaV/THqkPPH/i9kzO452zpKm3QQgn4rQtWICtZ7fScGZDfu74Mz2q9TA0izBTYiwsL2uamen5lUanAZ0KF1abWrUXVplai94dTIuGu+R4cpG81/pMfsI4ALe8pkLpUfj+r+6FwbPIP4U0/XZXT+v/zE8qwI/alstbJvO3EGnB2g+nbcEC1CtVD+983oSEh0iBtRcR35u6Rqt/aXQSE+UCJVuZHrdOmlq0J2bAuUdcdshR4J9C6PkM5Pd/cqF0L5y93c7mUi5p171zGJ1ECJvm1AXWRbnQ1a8rP+76ket3r1MwZ0GjI4kniY8xFdhSXaBQLaPTPCxPWag5Bqp+DjFbTC1JjyJpLc5CWb+VSAhhV5x2FPE9wVWCSUpNYmnEUqOjiKc5/A2k3IFqnxud5MncckLxFqYBUPn9TCv5SHEVwuk4fYGtU6IOZQqUkbmJbd2dKNPgpjKvQ/7KRqcRQoincvoCq5Qi2D+Y9SfXc+WOza0RL+459CWQClU+MTqJEEKYxekLLJi6iVN0CkuOLDE6iniUm5FwYrrpPso8vkanEUIIs0iBBao/U50KhStIN7GtOvip6ZYX/w+NTiKEEGaTAss/3cR/nf6LS7cuGR1HpHfjIJz+FSoOg5zFjE4jhBBmkwKbJtg/mFSdyqLDi4yOItIL+8g0G1Hl94xOIoQQGSIFNo1/UX/8vfylm9iWXNkJUcug8rum5eSEEMKOSIFNJ9g/mC1ntxAVJ4ts24QDH4KHl6l7WAgh7IwU2HSCqwQDsDB8ocFJBNF/wqU/wf/fMoetEMIuSYFNp0LhCtQsVlO6iY2mtan1mssbyg80Oo0QQmSKFNgHBPsHs/P8Tk7fOG10FOd1fgVc3WmaVMLaK8MIIYSVSIF9QFf/rgAsCF9gcBInpVMhbBTkLQ9lexqdRgghMk0K7APKFCxDYMlA6SY2ypkQ072vVT83TS4hhBB2SgrsIwT7B7P34l6OXz1udBTnkpoEYR9DgWpQuqvRaYQQIkukwD7Cy34vA0grNrudnAW3IqH6V2kLegshhP2S32KPUCp/KRr6NJQCm51S4uHgZ1A4CEq0NjqNEEJkmVkFVin1klLqqFIqUin1wWP2aaKU2q+UCldKbbJszOwX7B/MocuHOBxz2OgozuH4JLh7Hmp8DUoZnUYIIbLsqQVWKeUKTABaAn5Ad6WU3wP7FAAmAu201v7Ay1bImq26+HXBRbkQckhasVaXdBPCv4ZizeGZ541OI4QQFmFOCzYQiNRan9RaJwLzgfYP7PMKsERrfRZAa33ZsjGzX7E8xXiu9HOEhIegtTY6jmOLGAsJV6DaV0YnEUIIizGnwJYEzqV7HpX2WnoVgIJKqb+UUnuUUq8/6kBKqf5KqVClVGhMTEzmEmejYP9gjl49StilMKOjOK6EqxDxLXh3gCKBRqcRQgiLMafAPuqC2INNOjegNtAaeBH4SClV4aE3aT1Fax2gtQ7w8vLKcNjs1tmvM67KVQY7WdPh/5q6iKt9YXQSIYSwKHMKbBRQKt1zb+DCI/ZZrbW+rbW+AmwGqlsmonGK5CpCs7LNpJvYWu5cgGM/gm8PKFDF6DRCCGFR5hTY3UB5pVQZpZQ70A1Y/sA+y4BGSik3pVQuoC5wxLJRjRHsH8zJ6ycJvRBqdBTHE/6VaXKJqp8anUQIISzuqQVWa50MvAWswVQ0F2itw5VSA5VSA9P2OQKsBsKAXcA0rfUh68XOPh0rdSSHSw7pJra0W6fgxFR4ti/kfdboNEIIYXHKqK7PgIAAHRpqH63CtvPaciD6AKeHn8ZFZhiyjO294GwItI2EXA+OmRNCPI5Sao/WOsDoHOLppFqYIdg/mHNx59gRtcPoKI4h9jCcngvlB0txFUI4LCmwZmhXsR0erh4y6YSlhH0MrrnA75GTggkhhEOQAmuGfB75aFW+FQsPLyQlNcXoOPbtaiicWwyV3gHPIkanEUIIq5ECa6Zg/2Au3rrI32f/NjqKfQsbBe6FoPLbRicRQgirkgJrpjYV2pArRy4ZTZwVlzbBxTXg/y/Ikc/oNEIIYVVSYM2U2z03bSu0ZdHhRSSnJhsdx/5oDWEfQs4SpsFNQgjh4KTAZkCwfzAxd2LYeGqj0VHsz4VVELMVqnwEbjmNTiOEEFYnBTYDWpZvSV73vNJNnFE61dR6zV0GyvYxOo0QQmQLKbAZ4OnmSftK7VlyZAmJKYlGx7EfZxfB9f1Q7TNwdTc6jRBCZAspsBkU7B/M9fjrrD+53ugo9iE1GcI+gvx+UPoVo9MIIUS2kQKbQS2ebUEBzwLSTWyuU3Ph5jGo9iW4uBqdRgghso0U2Axyd3WnY6WOLI1YSnxyvNFxbFtKAhz8FAoFmBZUF0IIJyIFNhO6VelGXEIcayLXGB3FtkVOgTtnofrXoJTRaYQQIltJgc2EpmWaUiRXEeaHzzc6iu1Kvg3hX0LRJlCsudFphBAi20mBzQQ3Fze6VO7CsohlnLx+0ug4tunoeIi/DNW/ktarEMIpSYHNpH81+hfuru68uuRVmdnpQYnX4fB/oUQb8KpvdBohhDCEFNhM8snvw6TWk9getZ2vt3xtdBzbcuRbSLoB1b8wOokQQhhGCmwWdK/anR5Ve/D5ps9lMfZ77l6CiLHgEwwFaxidRgghDCMFNosmtJqAdz5vXl3yKjcTbhodx3jhX0NqAlT73OgkQghhKCmwWZTfMz9zO87l1I1TDF893Og4xrp9BiInQ9lekK+C0WmEEMJQUmAtoFHpRnzQ4ANm7J/BkiNLjI5jnINprdYqHxubQwghbIAUWAv5tMmnBJQIoN+KfpyPO290nOwXdxROzYLyb0JuH0Mi7Lu4j/rT68sEIEIImyAF1kJyuObgl06/EJ8cT69lvUjVqUZHyj6xR2D3IHDNCX7/MiTCvIPzaDCjAdujtvPhhg/RWhuSQwgh7pECa0EVClfghxd/YP3J9YzbMc7oONaVfBdOzoF1jeAPP7i82TQlYs5nsjVGSmoK7697n1eWvEJAiQA+b/I5ey7u4e+zf2drDiGEeJAUWAvrV6sf7Su254M/PyDsUpjRcSzvehiEDoHfSsCOnnA3GmqMhg5RUHFo9ka5e53Wv7ZmzLYxDAoYxPrX1/NO/XcolLMQP+z4IVuzCCHEg6TAWphSiqltp1IoZyF6LOnhGCvuJN2CE9NhTRCsqm6axL9ES2i2EdoeA7/3s73lGn45nMBpgWw4tYGpbacyofUE3F3dyZUjFwNqD2BpxFKZxlIIYSgpsFbglduLme1ncujyIT5Y/4HRcTLv2h7YNdDUWt3ZF5LioNYP0PECNPgVnmliyDzDSyOWEjQ9iFuJt/ir11/0rdX3vu2D6wzG1cWV8TvHZ3s2IYS4RwqslbxU7iWGBA5h3M5xrD2x1ug45kuKg+OTYVVtWB0Ap2ZDqU7wwt/QOhwqDQePwoZES9WpfPrXp3QM6UjlIpUJ7RdK/VIPz3VcMl9Jgv2Dmb5vOrHxsQYkFUIIKbBWNbr5aPy9/Om5tCdX7lwxOs7jaQ1XdsCOPrCkOOx+E3QyBPwEHS9CvVng1cDQVXFuJtyk84LOfLbpM3pW78nm3pspma/kY/cfETSCW4m3mL5vejamFEKIf0iBtaKcOXLyS6dfuHb3Gv1W9LO9W0cSr8PRH2FlNVhbD84uAN9XoMVOaLkfKgwG9wJGpyTyWiRB04NYcXQFY18cy8z2M/F083zie2qXqE0jn0aM3zleVjsSQhhCCqyVVS9Wna+bfs3SiKW20ZrSGi5vgW2vma6t7hkKrp4Q+H+m1mrdqVAk0GbWcF0duZo6U+tw6dYl1r62lmFBw1BmZhsRNIIzsWdYGrHUyimFEOJhZhVYpdRLSqmjSqlIpdRjR+0opeoopVKUUl0sF9H+jag3gmZlmjFs9TCOXz1uTIj4K3DkO9M9q+sbw/nlULY3vLQXXtoN5fpDjrzGZHsErTVjto6h9a+t8cnvw+5+u2lapmmGjtGuYjvKFiwrt+wIIQzx1AKrlHIFJgAtAT+gu1LK7zH7jQZknroHuCgXZnWYhYerBz2W9CApJSl7TqxTIfpP+LsbLC0J+94F94IQNNM0ErjORChUM3uyZMCdpDv0WNKD99e/T+fKndnWZxtlCpbJ8HFcXVwZGjiUbee2sev8LiskFUKIxzOnBRsIRGqtT2qtE4H5QPtH7DcEWAxctmA+h+Gdz5spbaew+8JuPt9k5aXc7kZD+DewogJsaA7Ra01zBLc6CC22mVa7cctt3QyZdObGGRrOaMj8Q/P5uunXhHQJIbd75rP2qdmHfB75pBUrhMh25hTYksC5dM+j0l77H6VUSaAjMPlJB1JK9VdKhSqlQmNiYjKa1e518etCrxq9+Prvr9l6dqtlD56aAhdWw5bOsLQUHPgX5PKGej+bWqu1x0KBKpY9p4VtOr2JgKkBnLh+ghXdV/CvRv8y+3rr4+T1yEvfmn1ZGL6Qc7Hnnv4GIYSwEHMK7KN+wz04HHYsMFJrnfKkA2mtp2itA7TWAV5eXuZmdCjjXxqPbwFfXv3tVeIS4rJ+wPjLcJKyCAMAACAASURBVPALWF4W/mppmhO40nBoEwHN/4IyPUyDmGyY1poJuybQfG5ziuQqwq6+u2hdobXFjj+k7hA0mp92/WSxYwohxNOYU2CjgFLpnnsDFx7YJwCYr5Q6DXQBJiqlOlgkoYPJ65GXnzv+zLnYcwxZNSRrB7sbbZq+8ODHpgXOG4SY5gSuOQbyVbRMYCtLSE6g7/K+vLXqLVqWa8mON3ZQsYhls/sW8KVT5U5M2TuFW4m3LHpsIYR4HHMK7G6gvFKqjFLKHegGLE+/g9a6jNbaV2vtCywCBmmt5d6Ix6hXqh6jGo9izoE5LAhfkLmDJMWZWqzxl+CFbdB0HZTuCq4elg1rRRdvXqTJ7CbM2D+DUY1GsbTbUvJ75rfKuUYEjeBG/A1m759tleMLIcSDnlpgtdbJwFuYRgcfARZorcOVUgOVUgOtHdBRjWo8irol6zLg9wEZvzaYkgCbO8GNQ9BoMXjVs05IK9oZtZPaU2pz8NJBFr68kC+afoGLst5t2fW86xFYMpBxO8c511q9QgjDmPUbTWu9UmtdQWv9rNb6q7TXJmutHxrUpLXupbVeZOmgjsbNxY2fO/1MUkoSPZf2NP+Xvk6F7T3h0p8QNANKvGTdoFYwc99MGs9qjKebJ9vf2E4XP+vfNq2UYkTQCI5fO84fx/6w+vmEEEJmcjJQuULlGN9yPBtPb+S7bd89/Q1aw9634WyIaQ3WMq9ZP6QFJaUkMXTVUPos70Mjn0bs7rebqs9Uzbbzd67cGe983nLLjhAiW0iBNVjvGr3pVLkTH274kH0X9z155yNj4Og4qDgMKr+XPQEtJOZ2DC1+bsGPu37k7aC3Wf3qagrnyt5VeXK45mBI4BA2nt7I/uj92XpuIYTzkQJrMKUUU9pMwSu3Fz2W9OBO0p1H73hyDuwfCT7BUOt7m5kr2Bz7o/dTZ2odtp/bzpwOc/juxe9wc3EzJEu/Wv3IlSMXY3eMNeT8QgjnIQXWBhTOVZhZ7Wdx5MoR3l/3/sM7XFgNO9+AZ5pCvdlgxcFAljb/0HzqT69Pik7h7z5/81p1Y7u1C+YsSO8avZl3aB7Rt6INzSKEcGz285vawb3w7AuMCBrBhN0TWHl85T8bru6Gv7uYZmFq/Jvd3IaTkprCB+s/oPvi7tQqXovQfqEElAgwOhYAw+oOIykliYm7JxodRQjhwKTA2pCvm31N1aJV6b2sN5dvX4a44/BXK/AoCk1WQY58Rkc0y434G7Sd15bRW0czoPYANvTcwDN5njE61v+UL1yeNhXaMCl0EneT7hodRwjhoKTA2hBPN09+7fwrsfGxvLOsB3rji6YNz6+GnMWMDWemwzGHCZwayPqT65ncejKT20zG3dXd6FgPGRE0git3rvDLwV+MjiKEcFBSYG1MlaJV+KHpZ7wdv57kO+fhuT9M0yDagWURywiaFkRsQiwbem5gQMAAoyM9VhPfJlR/pjpjd4xF6wen1hZCiKyTAmtrUhIYeHcdVT0UXS5CBLbfLRwVF8X7696nQ0gHKhSuQGi/UBr6NDQ61hPdm3giPCacdSfXGR1HCOGApMDaEp0KO3qhLv3JrZpj2Zqclx5LepCYkmh0sockpSSxLGIZbX5tQ+mxpRmzbQy9avRiS+8tlMpf6ukHsAHdqnSjWJ5ifL/9e6OjCCEckBRYW6E17H0HzsyHGqMp4DeUae2msffiXj7Z+InR6f7nxLUT/PvPf+Mz1ocOIR3Ye3EvHzT4gMghkcxsP5OcOXIaHdFsHm4eDK4zmDUn1nA45rDRcYQQDkYKrK048i0cHXvfLE0dKnWgb82+jN46mk2nNxkWLT45nnkH59FsTjPK/ViO0VtHU6dEHZZ1W8bZEWf5qtlXPFvoWcPyZcXAgIF4unnKxBNCCItTRg3wCAgI0KGhoYac2+acmgvbXzfN0tTg1/smkriVeIua/1eThOQEwt4Mo4BngWyLdejyIabtncbcsLlcu3sN3wK+9K3Zl141elEyX8lsy2Ft/Vf0Z27YXM6NOEeRXEWMjiPEEyml9mitbeOmcvFE0oI12oXVsKPPY2dpyuOeh186/cKFmxcY9Mcgq8e5lXiL6XunU296PapOqsrE3RNpXrY5615bx4mhJ/iw8YcOVVwBhgcNJz45nsmhDy0OJYQQmSYF1kj3ZmnK7//EWZoCSwbyaZNPmXdoHr+EWf6+Ta01u8/vpv+K/hT/rjh9V/QlNj6W71p8x4V3LhDSJYTmZZtbdb1WI/l5+fHisy8yYfcEEpITjI4jhHAQ0kVslLjjsK4BuOWGFtsgZ/En7p6SmsJzs57j4OWDHBh4AN8CvlmOcP3udX45+AtT904l7FIYOd1yElwlmH61+lHPux7KjhYUyKo1kWt46ZeXmN1hNq9Xf93oOEI8lnQR2w8psEa4Gw1r60PyTXhhq9kTSZy6forqk6tTo1gNNvbciKuLa4ZPrbVm85nNTNs3jUWHFxGfHE/t4rXpW6sv3at0J79n/gwf0xForakyqQruru7s7b/Xqf64EPZFCqz9cMw+P1uWFGeaXzj+UoZnaSpTsAwTWk1gy9ktjN46OkOnvXTrEv/d+l8q/lSRJrObsOLoCvrU6MPe/nsJ7R/KwICBTltcwTTxxPC6w9kfvZ9NZ4wbsS2EcBzSgs1OKYmm4np5Ezy3HEq0zPAhtNZ0X9ydxUcWs/2N7U9coSYlNYW1J9Yybd80lh9dTnJqMg19GtKvVj+6+HUhV45cWflpHM7dpLv4jPWhfqn6LOu2zOg4QjyStGDthzGrXjujtFmauPQnBM3OVHEFU0trUutJbDu3jR5LerC3/15yu+e+b5+zsWeZsW8GM/bN4FzcObxyeTG87nDeqPUGlYpUssAP45hy5sjJwNoD+WrLV0Rei6RcoXJGRxJC2DHpIs4O/5ulaR7U+AbKZm0QTcGcBZndYTbHrx7n7TVvA6apC5ccWULLX1riO9aXzzd9TmWvyix8eSFRb0cxpsUYKa5mGFRnEG4ubozbMc7oKEIIOyct2Oxw3yxN71vkkM+XeZ5367/LmG1juJt8lzUn1nD59mW883nzUeOP6F2zt0VGGjub4nmL071qd2bun8kXTb/I1ok9hBCORVqw1nZqLux/H3y6Qq3vwYKjU794/gtqFa/Frwd/pX6p+vzxyh+cHnaaz57/TIprFowIGsHtpNtM3TPV6ChCCDsmg5ys6cIa2NQGijaGJisfO5FEVtxKvEV8crxM8Wdhz89+nhPXTnBy2EncXKSjR9gOGeRkP6QFay1Xd8PfnZ86S1NW5XHPI8XVCkYEjeBc3DkWH15sdBQhhJ2SAmsNccfhr9bg4QXPr4Ictr9ourhfmwptKFeoHD/s+MHoKEIIOyUF1tLuRsPGFwENz6956hSIwja5KBeG1R3GzvM72X5uu9FxhBB2SAqsJWVhliZhe3rV6EUBzwLSihVCZIoUWEtJSYTNneBGGDRaBEUCjU4ksiiPex761erH4iOLOXPjjNFxhBB2RgqsJaSfpanu9EzP0iRsz5DAISgUP+760egoQgg7Y1aBVUq9pJQ6qpSKVEp98IjtPZRSYWmPbUqp6paPaqO0hr3vppulqafRiYQFlcpfii5+XZi6dyo3E24aHSdDYuNjWXdiHUbdiice7dKtS/LfxEk8tcAqpVyBCUBLwA/orpTye2C3U8BzWutqwBfAFEsHtVkR38HRH6DCUIvN0iRsy4igEcQlxDFz/0yjo5gt/HI4AVMDaPFzC9rMa8OlW5eMjiSA6FvRBEwN4MMNHxodRWQDc1qwgUCk1vqk1joRmA+0T7+D1nqb1vp62tMdgLdlY9qoUz/DvvdMszTV/sGiszQJ21HXuy71vOsxbuc4UlJTjI7zVIsPL6butLrcSrzFvxv+mz9P/knVSVX5/djvRkdzagnJCXQK6cS1u9fo6t/V6DgiG5hTYEsC59I9j0p77XHeAFY9aoNSqr9SKlQpFRoTE2N+Slt0cS3s6A3PPA/15oCSy9mObETQCE5eP8mKYyuMjvJYKakpfPjnh3RZ2IWqz1RlT/89fNXsK/b030PxvMVpO68tg/4YxJ2kO0ZHdTpaawb+MZDtUduZ3WE2NYrVMDqSyAbmVIVHNcseeQFBKfU8pgI78lHbtdZTtNYBWusALy8v81PamusHYEtnyO8Hjaw3S5OwHR0rd6R0/tI2e8vOjfgbtJ3Xlq///pq+NfvyV8+/KJG3BAD+Rf3Z1XcX79R7h0mhk6g9pTb7Lu4zOLFzGbtjLLP2z+KT5z6hi18Xo+OIbGJOgY0CSqV77g1ceHAnpVQ1YBrQXmt91TLxbNCdKNMsTTnym+YXds9vdCKRDdxc3BgSOITNZzaz9+Jeo+PcJ/xyOHWm1mH9yfVMaj2JKW2n4OF2/x99Hm4efNviW9a9to64hDjqTqvLf7f+l1SdalBq57Emcg3vrnuXTpU78fFzHxsdR2QjcwrsbqC8UqqMUsod6AYsT7+DUsoHWAK8prU+ZvmYNiIpzlRck+JMxTXXk3rKhaPpW6svedzz2FQrdsmRJQRND+Jmwk029tzIwICBqCeMBWhetjlhA8NoW7EtI9ePpPmc5pyLPffY/UXWHLt6jOBFwVQpWoXZHWbjIpeSnMpT/2trrZOBt4A1wBFggdY6XCk1UCk1MG23j4HCwESl1H6llOMtk5OaBFtehthw00QSBasZnUhks/ye+elTow/zD83nws2HOnGyVUpqCqM2jKLzgs74e/mzp/8eGvg0MOu9hXMVZtHLi5jebjq7zu+i2uRqLAxfaOXEzudet727qzvLuy0nj3seoyOJbGbWn1Na65Va6wpa62e11l+lvTZZaz057fu+WuuCWusaaQ/HWkpJa9j9JkSvhcApULyF0YmEQYbWHUpKagoTdk0wLMON+Bu0m9+Or7Z8xRs132BTr02UzJex3hSlFH1q9mH/wP1UKFyBrou60mtpL+IS4qyU2rmkpKbQbVE3Tl0/xeKuiyldoLTRkYQBpL/CHOFfw4np4D8Knu1jdBphoGcLPUv7Su2ZvGeyIaNx711vXXtiLRNbTWRq26kPXW/NiHKFyvF377/5qPFHzA2bS43JNWRxAwt4f937rDmxhomtJ9KodCOj4wiDSIF9mlO/QNgo8H0Vqn1udBphA0YEjeDa3WvMPTA3W8/74PXWN+u8+cTrrebK4ZqDz5//nM29NqPRNJrZiE//+pTk1GQLpHY+s/bP4vsd3zMkcAh9a/U1Oo4wkBTYJ7n0F+zsDUWbmOYYlokkBNDIpxG1itdi7M6x2TIKN1Wn8tGGj+673trQp6HFz9PApwH7B+znlaqv8Nmmz2g0sxEnrp2w+Hkc2fZz2xnw+wCalWnG9y9+b3QcYTApsI8TewQ2d4Q85aDxEnB1NzqRsBFKKUYEjSDiSgRrItdY9Vz3Bsp8ueXLTF9vzYj8nvmZ03EO8zvP50jMEWr8Xw1m7Z8lc+ea4VzsOTqGdKRUvlIseHkBbi5uRkcSBpMC+yh3o+GvlqYJJJqsBPeCRicSNqarf1eK5ylu1Vt2DsccJnBqoMWut2ZEcJVgwt4Mo3bx2vRe1puui7py7e61bDm3PbqTdIcOIR24k3SH5d2XUyhnIaMjCRsgBfZBybdhUxuIj4Hnfoc8vkYnEjbI3dWdtwLfYt3JdRy6fMjix//tyG/UnVaXuIQ4i15vzQif/D78+fqffNPsG5ZGLKXapGpsOLUhWzPYA601fZb1Yd/FfczrPA8/rwfXQhHOSgpseqkpsLU7XN8HDUOgsGPdbSQsa0DtAeR0y8nYHWMtdsxUncrHGz+m04JO+Hn5Edo/1CrXW83l6uLKyIYj2fHGDvK456H5nOa8v+59EpITDMtka77e8jUh4SF80/wbWldobXQcYUOkwN6jNewZBudXQO0foWQboxMJG1c4V2F6Vu/Jz2E/c/n25Swf70b8DdrNa8cXm7+gT40+bOq1Ce98trEwVe0StdnTfw8Dag9gzLYxBE0P4kjMEaNjGW5ZxDJGbRzFq9Ve5b367xkdR9gYKbD3RHwPxydA5XehwiCj0wg7MTxoOAkpCUwOnZyl49y73rrmxBomtJrAtHbT8HTztFBKy8jtnptJbSaxrNsyouKiqDWlFhN3T3TaAVAHLx2kx5IeBJYMZGrbqdnehS9snxRYgLOLYN+7UKoL1BhtdBphRyoWqUir8q2YuHtiprtNl0Yspe60usQmxLLh9Q0MqjPIpn9Zt6vYjoNvHqSJbxMGrxxM23ltnW5B9yt3rtBufjvyeeTjt+DfbO6PIWEbpMDGbINtr0KRerKuq8iUEUEjuHT7EvMOzcvQ++5db+0Y0pHKRSqzp/8eu5n1p1ieYqx8ZSXjXxrP+pPrqTa5Gn8c+8PoWNkiMSWRLgu6cPHmRZZ2W/q/ZQGFeJBzV5O447C5HeQqBY2Xg1tOoxMJO9SsTDOqFq3KDzt+MLu7NDY+lvbz2/PF5i/oXaM3m3tvtpnrreZSSjGk7hD29N9DsTzFaDOvDYP/GOzwC7oPWzWMTWc2Mb3ddAJLBhodR9gw5y2w8Vfgr1am759fBZ5FjM0j7JZSiuFBwwm7FMbG0xufuv+RmCMETgtkdeRqfmr5E9PbTbfrLsZ7C7q/HfQ2E0MnEjAlwGEXdJ+0exKT90xmZIOR9KjWw+g4wsY5Z4FNvgub28Odc6aWa95yRicSdu6Vqq9QNHfRp048ce966434G/z5+p8MDhxs09dbzeXh5sF3L37HutfWEZsQS91pdRmzdYxDLei+8dRGhqwaQuvyrfmq6VdGxxF2wPkKrE6FHT3hynao/zN41Tc6kXAAnm6evBnwJr8f+51jV489tD1Vp/LJxk/oGNKRSkUqEdovlMalGxuQ1LrSL+j+/vr3eWHuC0TFRRkdK8tOXj9Jl4VdqFC4Ar92/hVXF1ejIwk74HwFdv9IOLsQao4Bny5GpxEO5M2AN3F3dWfcjnH3vR4bH0uH+R34fPPn9KrRi829N1MqfymDUlpf+gXdd0btpNqkavx68Fe7bc3eTLhJu3nt0FqzvPty8nnkMzqSsBPOVWCPTYQj30L5wVDpbaPTCAfzTJ5n6FG1B7MOzPrfvL0RVyKoO60uqyJX8VPLn5jRboZdX281170F3fcN2Ef5wuXpsaQHdabWYdXxVXZ132yqTuXV314l4koEC19eSLlCcjlJmM95CmzUCtgzBEq2hdrjZOk5YRUjgkZwJ+kOU/dMZVnEMgKnBnLt7jWHut6aEeULl2dbn23M6TCHG/E3aPVrKxrNbMSm05uMjmaWjzZ8xPKjyxn70lialW1mdBxhZ5RRf00GBATo0NDQ7DnZ1VBY/xzk94Pmf4Fb7uw5r3BKzec0Z+f5ndxKvEVAiQCWdF3i0F3C5kpKSWLGvhl8vvlzLty8wAtlX+Crpl9Rp2Qdo6M90ryD83hlySv0q9WP/2vzfzbzx5FSao/WWiZKtwOO34K9ddq0Oo6nFzy3QoqrsLr36r/HrcRb9KrRiy29t0hxTZPDNQcDAgYQOSSS71p8x77ofQROC6TD/A4cvHTQ6Hj3Cb0QSp/lfWjk04ifWv1kM8VV2BfHbsEmXoe1DeDuRWixDfJXtu75hEhz8eZFiuUpJr+Yn+Bmwk3G7RzHt9u+JS4hjm5VuvFZk88oX7i8obku3rxInal1cHNxY3e/3Xjl9jI0z4OkBWs/HLcFm5IAmzvBrUhovFSKq8hWxfMWl+L6FHk98jKq8ShODjvJBw0/YNnRZVSeUJm+y/tyNvasIZnik+PpGNKRG/E3WNZtmc0VV2FfHLPAag07+8Llv6DuTHjmOaMTCSEeo1DOQnzd7GtODj3JW4FvMTdsLuV/LM/QVUOJvhWdbTm01vRf0Z+d53cyp+Mcqhernm3nFo7JMQts2Mdw+meo9iWUkenMhLAHz+R5hrEvjSVySCQ9q/dk4u6JPDv+Wf61/l//u+3Jmr7b/h1zw+byWZPP6FS5k9XPJxyf412DPTEDdr4Bz74BgVPldhwh7NTxq8f5dNOnzDs4j7weeXmn3juMCBpBXo+8Fj/XyuMrafNrG7r4dSGkS4hNd+/LNVj74Vgt2ItrYVd/KNYC6kyS4iqEHStfuDy/dPqFsDfDaFqmKZ/89QllxpXhu23fcTfprsXOE3Elgu6Lu1O9WHVmtp9p08VV2BfHKbDXD8CWLpDfHxotBJccRicSQlhAlaJV+C34N3b13UXtErV5d927lPuxHJN2TyIxJTFLx75+9zrt5rXD082TZd2WkdtdbuMTluMYBfZOFPzVGnLkgyZ/mL4KIRxKnZJ1WPPqGjb12kTZgmUZtHIQFX+qyOz9s0lJTcnw8ZJTkwleFMzpG6dZ0nUJPvl9rJBaODP7L7BJcabimhRnKq657GvRaiFExjQu3ZjNvTazqscqCuUsRK9lvagyqQoLwxdmaEGBd9e+y7qT65jcZjINfBpYMbFwVvZdYFOT4O+uEBsODRdCQRlWL4QzUErxUrmXCO0XyqKXF6FQdF3UldpTavPHsT+euqDA9L3TGbdzHMPqDqNPzT7ZlFo4G/stsFrD7kFwcQ0E/h+UeNHoREKIbKaUorNfZw6+eZA5HeYQlxBHm3ltaDizIRtPbXzke7ae3cqbf7zJC2Vf4NsW32ZzYuFMzCqwSqmXlFJHlVKRSqkPHrFdKaXGp20PU0rVsnzUBxz+D5yYBv4fmm7JEUI4LVcXV16r/hoRgyOY3HoyZ26coemcpqaFF6J2/m+/s7Fn6bSgE74FfAnpEoKbi5uBqYWje2qBVUq5AhOAloAf0F0p5ffAbi2B8mmP/sAkC+e83+lf4cCH4NsDqn1h1VMJIezHvQUFjg85zvctvifsUhhB04NoN68d289tp/389sQnx7O8+3IK5ixodFzh4MxpwQYCkVrrk1rrRGA+0P6BfdoDc7TJDqCAUqq4hbOaXNoEO3pD0SZQd7rc6yqEeEjOHDkZUW8EJ4ed5Mvnv2Tzmc3Un1GfA9EHmN95PpWKVDI6onAC5vSPlATOpXseBdQ1Y5+SwMX0Oyml+mNq4eLjk8kh8R6F4ZnnocE8cPXI3DGEEE4hj3sePmz8IYPqDOLHXT9SpkAZWpZvaXQs4STMKbCPaiI+OETPnH3QWk8BpoBpqkQzzv2wAlXg+dWZeqsQwjkVzFmQj5/72OgYwsmY00UcBaRfMdobuJCJfYQQQginYU6B3Q2UV0qVUUq5A92A5Q/ssxx4PW00cRAQq7W++OCBhBBCCGfx1C5irXWyUuotYA3gCszQWocrpQambZ8MrARaAZHAHaC39SILIYQQts+sm8C01isxFdH0r01O970GBls2mhBCCGG/7HcmJyGEEMKGSYEVQgghrEAKrBBCCGEFUmCFEEIIK1BPW9bJaidWKgY4k8m3FwGuWDCOvZPP437yefxDPov7OcLnUVpr7WV0CPF0hhXYrFBKhWqtA4zOYSvk87iffB7/kM/ifvJ5iOwkXcRCCCGEFUiBFUIIIazAXgvsFKMD2Bj5PO4nn8c/5LO4n3weItvY5TVYIYQQwtbZawtWCCGEsGlSYIUQQggrsKsCq5QqpZTaqJQ6opQKV0oNMzqT0ZRSrkqpfUqp343OYjSlVAGl1CKlVETav5F6RmcyklJqRNr/J4eUUvOUUp5GZ8pOSqkZSqnLSqlD6V4rpJRap5Q6nva1oJEZhWOzqwILJAPvaK0rA0HAYKWUn8GZjDYMOGJ0CBsxDlitta4EVMeJPxelVElgKBCgta6CaanJbsamynazgJceeO0D4E+tdXngz7TnQliFXRVYrfVFrfXetO9vYvoFWtLYVMZRSnkDrYFpRmcxmlIqH9AYmA6gtU7UWt8wNpXh3ICcSik3IBdwweA82UprvRm49sDL7YHZad/PBjpkayjhVOyqwKanlPIFagI7jU1iqLHA+0Cq0UFsQFkgBpiZ1mU+TSmV2+hQRtFanwe+Bc4CF4FYrfVaY1PZhGe01hfB9Ac7UNTgPMKB2WWBVUrlARYDw7XWcUbnMYJSqg1wWWu9x+gsNsINqAVM0lrXBG7jxN1/adcW2wNlgBJAbqXUq8amEsK52F2BVUrlwFRcf9FaLzE6j4EaAO2UUqeB+UBTpdTPxkYyVBQQpbW+16OxCFPBdVbNgVNa6xitdRKwBKhvcCZbcEkpVRwg7etlg/MIB2ZXBVYppTBdYzuitf7e6DxG0lr/S2vtrbX2xTR4ZYPW2mlbKFrraOCcUqpi2kvNgMMGRjLaWSBIKZUr7f+bZjjxoK90lgM9077vCSwzMItwcG5GB8igBsBrwEGl1P601/6ttV5pYCZhO4YAvyil3IGTQG+D8xhGa71TKbUI2Itp9P0+nGyaQKXUPKAJUEQpFQV8AnwDLFBKvYHpj5CXjUsoHJ1MlSiEEEJYgV11EQshhBD2QgqsEEIIYQVSYIUQQggrkAIrhBBCWIEUWCGEEMIKpMAKh6GU8k2/coqtHlMI4RykwAohhBBWIAVWOCSlVNm0Sf/rPPB6iFKqVbrns5RSndNaqluUUnvTHg9NK6iU6qWU+ind89+VUk3Svm+hlNqe9t6FafNlCyGcmBRY4XDSpktcDPTWWu9+YPN8IDhtP3dMUwiuxDQn7Qta61pp28dn4HxFgFFA87T3hwJvZ/XnEELYN3ubKlGIp/HCNL9sZ611+CO2rwLGK6U8MC3GvVlrfVcplR/4SSlVA0gBKmTgnEGAH7DVNO0v7sD2LPwMQggHIAVWOJpY4BymeasfKrBa63il1F/Ai5haqvPSNo0ALgHVMfXsxD/i2Mnc3+vjmfZVAeu01t0tkF8I4SCki1g4mkSgA/C6UuqVx+wzH9NCAI2ANWmv5Qcuaq1TMS0o4fqI950GaiilXJRSpYDAtNd3AA2UUuUA0lawyUgLWAjhgKTACoejtb4NtAFGKKXaP2KXtUBjYL3Wp2Q6VwAAAHFJREFUOjHttYlAT6XUDkzdw7cf8b6twCngIPAtppVq0FrHAL2AeUqpMEwFt5LFfiAhhF2S1XSEEEIIK5AWrBBCCGEFUmCFEEIIK5ACK4QQQliBFFghhBDCCqTACiGEEFYgBVYIIYSwAimwQgghhBX8PwztfL1Y0Ut0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Get the result of k-fold cross validation\n",
    "rmses_normalized = []\n",
    "stds_normalized = []\n",
    "x = range(2, 12)\n",
    "\n",
    "#RMSE\n",
    "for i in rmses[2:]:\n",
    "    rmses_normalized.append((i - min(rmses[2:]))/(max(rmses[2:]) - min(rmses[2:])))\n",
    "\n",
    "plt.plot(x,rmses_normalized, label = 'RMSE', color='green')\n",
    "plt.xlabel('k value')\n",
    "\n",
    "#STD\n",
    "for i in stds[2:]:\n",
    "    stds_normalized.append((i - min(stds[2:]))/(max(stds[2:]) - min(stds[2:])))\n",
    "\n",
    "plt.plot(x,stds_normalized, label = 'STD', color='orange')\n",
    "plt.legend(bbox_to_anchor=(1.3,1), borderaxespad=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the graph shows there is a tradeoff between RMSE and STD. A k-value of 4 looks optimal from the graph though. We can verify this by selecting the k-value with the lowest combined standardized scores for average RMSE and STD of RMSE (here we assume we care about RMSE and STD equally). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The optimal k-value is: 4 which has a combined standardized score of: 0.6419133337948719\n"
     ]
    }
   ],
   "source": [
    "# A min-max scaling is used to standardize both average RMSE and std of RMSE values. \n",
    "\n",
    "combined = []\n",
    "for a,b in zip(rmses_normalized, stds_normalized):\n",
    "    combined.append(a+b)\n",
    "\n",
    "def optimum(combined_rmse_std_list):\n",
    "    k_value = 2\n",
    "    base_value = combined_rmse_std_list[0]\n",
    "    iteration = 2\n",
    "    for c in combined_rmse_std_list:\n",
    "        if c < base_value:\n",
    "            base_value = c\n",
    "            k_value = iteration\n",
    "            iteration = iteration + 1\n",
    "        else:\n",
    "            iteration = iteration + 1\n",
    "    \n",
    "    return k_value, base_value\n",
    "        \n",
    "\n",
    "optimum(combined)\n",
    "\n",
    "print(\"The optimal k-value is:\", str(optimum(combined)[0]), \"which has a combined standardized score of:\", str(optimum(combined)[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project used feature engineering and selection to create and select features to train a multiple linear regression model. Hold out, simple and k-fold cross validation was used to validate the model. \n",
    "\n",
    "When using cross-validation (where k>1) to validate the regression model, the optimal number of folds is 4. \n",
    "\n",
    "We can then use our regression model (and optimal number of folds) to predict house sale prices in the city of Ames, Iowa if we know its features. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}