african_wells_solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<style>\n",
    "@font-face {\n",
    "  font-family: CharisSILW;\n",
    "  src: url(files/CharisSIL-R.woff);\n",
    "}\n",
    "@font-face {\n",
    "  font-family: CharisSILW;\n",
    "  font-style: italic;\n",
    "  src: url(files/CharisSIL-I.woff);\n",
    "}\n",
    "@font-face {\n",
    "\tfont-family: CharisSILW;\n",
    "\tfont-weight: bold;\n",
    "\tsrc: url(files/CharisSIL-B.woff);\n",
    "}\n",
    "@font-face {\n",
    "\tfont-family: CharisSILW;\n",
    "\tfont-weight: bold;\n",
    "\tfont-style: italic;\n",
    "\tsrc: url(files/CharisSIL-BI.woff);\n",
    "}\n",
    "\n",
    "div.cell, div.text_cell_render{\n",
    "    max-width:1000px;\n",
    "}\n",
    "\n",
    "h1 {\n",
    "    text-align:center;\n",
    "    font-family: Charis SIL, CharisSILW, serif;\n",
    "}\n",
    "\n",
    ".rendered_html {\n",
    "    font-size: 130%;\n",
    "    line-height: 1.3;\n",
    "}\n",
    "\n",
    ".rendered_html li {\n",
    "    line-height: 2;\n",
    "}\n",
    "\n",
    ".rendered_html h1{\n",
    "    line-height: 1.3;\n",
    "}\n",
    "\n",
    ".rendered_html h2{\n",
    "    line-height: 1.2;\n",
    "}\n",
    "\n",
    ".rendered_html h3{\n",
    "    line-height: 1.0;\n",
    "}\n",
    "\n",
    ".text_cell_render {\n",
    "    font-family: Charis SIL, CharisSILW, serif;\n",
    "    line-height: 145%;\n",
    "}\n",
    "\n",
    "li li {\n",
    "    font-size: 85%;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End-to-End Data Science in Python\n",
    "\n",
    "<img src=\"scikit-learn.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    " This is the workbook for the \"End-to-End Data Analysis in Python\" workshop\n",
    " at the Open Data Science Conference 2015, in beautiful San Francisco. \n",
    " This notebook contains starter code only; the goal is that we will fill in the\n",
    " gaps together as we progress through the workshop.  If, however, you're doing this\n",
    " asynchronously or you get stuck, you can reference the solutions workbook.\n",
    "\n",
    " The objective is to complete the \"Pump it Up: Mining the Water Table\" challenge\n",
    " on [drivendata.org](www.drivendata.org/competitions/7/); the objective here is to predict\n",
    " African wells that are non-functional or in need of repair.  Per the rules of the\n",
    " competition, you should register for an account with drivendata.org, at which point you\n",
    " can download the training set values and labels.  We will be working with those datasets\n",
    " during this workshop.  You should download those files to the directory in which this\n",
    " notebook lives, and name them wells_features.csv and wells_labels.csv (to be consistent\n",
    " with our nomenclature).  You are also encouraged to continue developing your solution\n",
    " after this workshop, and/or to enter your solution in the competition on the drivendata\n",
    " website!\n",
    " \n",
    " ### Code requirements\n",
    " Here's the environment you'll need to work with this code base:\n",
    "\n",
    " * python 3 (2.x may work with minor changes, but no guarantees)\n",
    " * pandas\n",
    " * scikit-learn\n",
    " * numpy\n",
    "\n",
    "# First Draft of an Analysis\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                  status_group\n",
      "id                            \n",
      "69572               functional\n",
      "8776                functional\n",
      "34310               functional\n",
      "67743           non functional\n",
      "19728               functional\n",
      "9944                functional\n",
      "19816           non functional\n",
      "54551           non functional\n",
      "53934           non functional\n",
      "46144               functional\n",
      "49056               functional\n",
      "50409               functional\n",
      "36957               functional\n",
      "50495               functional\n",
      "53752               functional\n",
      "61848               functional\n",
      "48451           non functional\n",
      "58155           non functional\n",
      "34169  functional needs repair\n",
      "18274               functional\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "  \n",
    "features_df = pd.DataFrame.from_csv(\"well_data.csv\")\n",
    "labels_df   = pd.DataFrame.from_csv(\"well_labels.csv\")  \n",
    "print( labels_df.head(20) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " One nice feature of ipython notebooks is it's easy to make small changes to code and\n",
    " then re-execute quickly, to see how things change.  For example, printing the first 5 lines\n",
    " of the labels dataframe (which is the default) isn't really ideal here, since there's a label\n",
    " (\"functional needs repair\") which doesn't appear in the first five lines.  Type 20 in the\n",
    " parentheses labels_df.head(), so it now reads labels_df.head(20), and press shift-enter to\n",
    " rerun the code.  See the difference?\n",
    " \n",
    " Now take a quick look at the features, again by calling .head() (set up for you in the code box\n",
    " below, or add your own code to the code box above).  You can print or as few\n",
    " rows as you like.  Take a quick look at the data--approximately how many features are there?\n",
    " Are they all numeric, or will you have to do work to transform non-numeric features into\n",
    " numbers?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       amount_tsh date_recorded        funder  gps_height     installer  \\\n",
      "id                                                                        \n",
      "69572        6000       3/14/11         Roman        1390         Roman   \n",
      "8776            0        3/6/13       Grumeti        1399       GRUMETI   \n",
      "34310          25       2/25/13  Lottery Club         686  World vision   \n",
      "67743           0       1/28/13        Unicef         263        UNICEF   \n",
      "19728           0       7/13/11   Action In A           0       Artisan   \n",
      "\n",
      "       longitude   latitude              wpt_name  num_private  \\\n",
      "id                                                               \n",
      "69572  34.938093  -9.856322                  none            0   \n",
      "8776   34.698766  -2.147466              Zahanati            0   \n",
      "34310  37.460664  -3.821329           Kwa Mahundi            0   \n",
      "67743  38.486161 -11.155298  Zahanati Ya Nanyumbu            0   \n",
      "19728  31.130847  -1.825359               Shuleni            0   \n",
      "\n",
      "                         basin          ...          payment_type  \\\n",
      "id                                      ...                         \n",
      "69572               Lake Nyasa          ...              annually   \n",
      "8776             Lake Victoria          ...             never pay   \n",
      "34310                  Pangani          ...            per bucket   \n",
      "67743  Ruvuma / Southern Coast          ...             never pay   \n",
      "19728            Lake Victoria          ...             never pay   \n",
      "\n",
      "      water_quality  quality_group      quantity quantity_group  \\\n",
      "id                                                                \n",
      "69572          soft           good        enough         enough   \n",
      "8776           soft           good  insufficient   insufficient   \n",
      "34310          soft           good        enough         enough   \n",
      "67743          soft           good           dry            dry   \n",
      "19728          soft           good      seasonal       seasonal   \n",
      "\n",
      "                     source           source_type source_class  \\\n",
      "id                                                               \n",
      "69572                spring                spring  groundwater   \n",
      "8776   rainwater harvesting  rainwater harvesting      surface   \n",
      "34310                   dam                   dam      surface   \n",
      "67743           machine dbh              borehole  groundwater   \n",
      "19728  rainwater harvesting  rainwater harvesting      surface   \n",
      "\n",
      "                   waterpoint_type waterpoint_type_group  \n",
      "id                                                        \n",
      "69572           communal standpipe    communal standpipe  \n",
      "8776            communal standpipe    communal standpipe  \n",
      "34310  communal standpipe multiple    communal standpipe  \n",
      "67743  communal standpipe multiple    communal standpipe  \n",
      "19728           communal standpipe    communal standpipe  \n",
      "\n",
      "[5 rows x 39 columns]\n"
     ]
    }
   ],
   "source": [
    "print( features_df.head() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming string labels into integers\n",
    " The machine learning algorithms downstream are not going to handle it well if the class labels\n",
    " used for training are strings; instead, we'll want to use integers.  The mapping that we'll use\n",
    " is that \"non functional\" will be transformed to 0, \"functional needs repair\" will be 1, and\n",
    " \"functional\" becomes 2.\n",
    " \n",
    " There are a number of ways to do this; the framework below uses applymap() in pandas.\n",
    " [Here's](http://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.applymap.html)\n",
    " the documentation for applymap(); in the code below, you should fill in the function body for\n",
    " label_map(y) so that if y is \"functional\", label_map returns 2; if y is \"functional needs\n",
    " repair\" then it should return 1, and \"non functional\" is 0. \n",
    " There's a print statement there to help you confirm that the label transformation is working\n",
    " properly.\n",
    " \n",
    " As an aside, you could also use apply() here if you like.  The difference between apply()\n",
    " and applymap() is that applymap() operates on a whole dataframe while apply() operates on a series\n",
    " (or you can think of it as operating on one column of your dataframe).  Since labels_df only has\n",
    " one column (aside from the index column), either one will work here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       status_group\n",
      "id                 \n",
      "69572             2\n",
      "8776              2\n",
      "34310             2\n",
      "67743             0\n",
      "19728             2\n"
     ]
    }
   ],
   "source": [
    "def label_map(y):\n",
    "    if y==\"functional\":\n",
    "        return 2\n",
    "    elif y==\"functional needs repair\":\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "labels_df = labels_df.applymap(label_map)\n",
    "print( labels_df.head() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming string features into integers\n",
    " \n",
    "Now that the labels are ready, we'll turn our attention to the features.  Many of the features\n",
    "are categorical, where a feature can take on one of a few discrete values, which are not ordered.\n",
    "Fill in the function body of ``transform_feature( df, column )`` below so that it takes our ``features_df`` and\n",
    "the name of a column in that dataframe, and returns the same dataframe but with the indicated\n",
    "feature encoded with integers rather than strings.\n",
    "\n",
    "We've provided code to wrap your transformer function in a loop iterating through all the columns that should\n",
    "be transformed.\n",
    "\n",
    "Last, add a line of code at the bottom of the block below that removes the ``date_recorded`` column from ``features_df``.  Time-series information like dates and times need special treatment, which we won't be going into today."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       amount_tsh date_recorded  funder  gps_height  installer  longitude  \\\n",
      "id                                                                          \n",
      "69572        6000       3/14/11     614        1390        685  34.938093   \n",
      "8776            0        3/6/13    1206        1399       1252  34.698766   \n",
      "34310          25       2/25/13     878         686        555  37.460664   \n",
      "67743           0       1/28/13     920         263       2059  38.486161   \n",
      "19728           0       7/13/11     906           0        164  31.130847   \n",
      "\n",
      "        latitude  wpt_name  num_private  basin          ...            \\\n",
      "id                                                      ...             \n",
      "69572  -9.856322     15454            0      5          ...             \n",
      "8776   -2.147466     19453            0      8          ...             \n",
      "34310  -3.821329     10040            0      0          ...             \n",
      "67743 -11.155298     19651            0      7          ...             \n",
      "19728  -1.825359      7904            0      8          ...             \n",
      "\n",
      "       payment_type  water_quality  quality_group  quantity  quantity_group  \\\n",
      "id                                                                            \n",
      "69572             0              2              3         4               4   \n",
      "8776              2              2              3         2               2   \n",
      "34310             6              2              3         4               4   \n",
      "67743             2              2              3         3               3   \n",
      "19728             2              2              3         1               1   \n",
      "\n",
      "       source  source_type  source_class  waterpoint_type  \\\n",
      "id                                                          \n",
      "69572       3            0             2                3   \n",
      "8776        8            4             0                3   \n",
      "34310       7            3             0                2   \n",
      "67743       0            2             2                2   \n",
      "19728       8            4             0                3   \n",
      "\n",
      "       waterpoint_type_group  \n",
      "id                            \n",
      "69572                      2  \n",
      "8776                       2  \n",
      "34310                      2  \n",
      "67743                      2  \n",
      "19728                      2  \n",
      "\n",
      "[5 rows x 39 columns]\n",
      "['amount_tsh' 'funder' 'gps_height' 'installer' 'longitude' 'latitude'\n",
      " 'wpt_name' 'num_private' 'basin' 'subvillage' 'region' 'region_code'\n",
      " 'district_code' 'lga' 'ward' 'population' 'public_meeting' 'recorded_by'\n",
      " 'scheme_management' 'scheme_name' 'permit' 'construction_year'\n",
      " 'extraction_type' 'extraction_type_group' 'extraction_type_class'\n",
      " 'management' 'management_group' 'payment' 'payment_type' 'water_quality'\n",
      " 'quality_group' 'quantity' 'quantity_group' 'source' 'source_type'\n",
      " 'source_class' 'waterpoint_type' 'waterpoint_type_group']\n"
     ]
    }
   ],
   "source": [
    "def transform_feature( df, column_name ):\n",
    "    unique_values = set( df[column_name].tolist() )\n",
    "    transformer_dict = {}\n",
    "    for ii, value in enumerate(unique_values):\n",
    "        transformer_dict[value] = ii\n",
    "\n",
    "    def label_map(y):\n",
    "        return transformer_dict[y]\n",
    "    df[column_name] = df[column_name].apply( label_map )\n",
    "    return df\n",
    "\n",
    "### list of column names indicating which columns to transform; \n",
    "### this is just a start!  Use some of the print( labels_df.head() )\n",
    "### output upstream to help you decide which columns get the\n",
    "### transformation\n",
    "names_of_columns_to_transform = [\"funder\", \"installer\", \"wpt_name\", \"basin\", \"subvillage\",\n",
    "                    \"region\", \"lga\", \"ward\", \"public_meeting\", \"recorded_by\",\n",
    "                    \"scheme_management\", \"scheme_name\", \"permit\",\n",
    "                    \"extraction_type\", \"extraction_type_group\",\n",
    "                    \"extraction_type_class\",\n",
    "                    \"management\", \"management_group\",\n",
    "                    \"payment\", \"payment_type\",\n",
    "                    \"water_quality\", \"quality_group\", \"quantity\", \"quantity_group\",\n",
    "                    \"source\", \"source_type\", \"source_class\",\n",
    "                    \"waterpoint_type\", \"waterpoint_type_group\"]\n",
    "for column in names_of_columns_to_transform:\n",
    "    features_df = transform_feature( features_df, column )\n",
    "    \n",
    "print( features_df.head() )\n",
    "    \n",
    "### remove the \"date_recorded\" column--we're not going to make use\n",
    "### of time-series data today\n",
    "features_df.drop(\"date_recorded\", axis=1, inplace=True)\n",
    "\n",
    "print(features_df.columns.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, a couple last steps to get everything ready for sklearn.  The features and labels are taken out of their dataframes and put into a numpy.ndarray and list, respectively.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "X = features_df.as_matrix()\n",
    "y = labels_df[\"status_group\"].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Predicting well failures with logistic regression\n",
    "\n",
    "The cheapest and easiest way to train on one portion of your dataset and test on another, and to get a measure of model quality at the same time, is to use ``sklearn.cross_validation.cross_val_score()``.  This splits your data into 3 equal portions, trains on two of them, and tests on the third.  This process repeats 3 times.  That's why 3 numbers get printed in the code block below.\n",
    "\n",
    "You don't have to add anything to the code block, it's ready to go already.  However, use it for reference in the next part of the tutorial, where you will be looking at other sklearn algorithms.\n",
    "\n",
    "Heads up: it can be a little slow.  This took a minute or two to evaluate on my MacBook Pro."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.65363636  0.6569697   0.6560101 ]\n"
     ]
    }
   ],
   "source": [
    "import sklearn.linear_model\n",
    "import sklearn.cross_validation\n",
    "\n",
    "clf = sklearn.linear_model.LogisticRegression()\n",
    "score = sklearn.cross_validation.cross_val_score( clf, X, y )\n",
    "print( score )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Comparing logistic regression to tree-based methods\n",
    "\n",
    "We have a baseline logistic regression model for well failures.  Let's compare to a couple of other classifiers, a decision tree classifier and a random forest classifier, to see which one seems to do the best.  \n",
    "\n",
    "Code this up on your own.  You can use the code in the box above as a kind of template, and just drop in the new classifiers.  The sklearn documentation might also be helpful:\n",
    "* [Decision tree classifier](http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html)\n",
    "* [Random forest classifier](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)\n",
    "\n",
    "We will talk about all three of these models more in the next part of the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.73590909  0.73691919  0.73005051]\n",
      "[ 0.78777778  0.7889899   0.78409091]\n",
      "['amount_tsh' 'funder' 'gps_height' 'installer' 'longitude' 'latitude'\n",
      " 'wpt_name' 'num_private' 'basin' 'subvillage' 'region' 'region_code'\n",
      " 'district_code' 'lga' 'ward' 'population' 'public_meeting' 'recorded_by'\n",
      " 'scheme_management' 'scheme_name' 'permit' 'construction_year'\n",
      " 'extraction_type' 'extraction_type_group' 'extraction_type_class'\n",
      " 'management' 'management_group' 'payment' 'payment_type' 'water_quality'\n",
      " 'quality_group' 'quantity' 'quantity_group' 'source' 'source_type'\n",
      " 'source_class' 'waterpoint_type' 'waterpoint_type_group']\n"
     ]
    }
   ],
   "source": [
    "import sklearn.tree\n",
    "import sklearn.ensemble\n",
    "\n",
    "clf = sklearn.tree.DecisionTreeClassifier()\n",
    "score = sklearn.cross_validation.cross_val_score( clf, X, y )\n",
    "print( score )\n",
    "\n",
    "clf = sklearn.ensemble.RandomForestClassifier()\n",
    "score = sklearn.cross_validation.cross_val_score( clf, X, y )\n",
    "print( score )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations!  You have a working data science setup, in which you have:\n",
    "* read in data\n",
    "* transformed features and labels to make the data amenable to machine learning\n",
    "* made a train/test split (this was done implicitly when you called ``cross_val_score``)\n",
    "* evaluated several models for identifying wells that are failed or in danger of failing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Paying down technical debt and tuning the models\n",
    "\n",
    "We got things running really fast, which is great, but at the cost of being a little quick-and-dirty about some details.  First, we got the features encoded as integers, but they really should be dummy variables.  Second, it's worth going through the models a little more thoughtfully, to try to understand their performance and if there's any more juice we can get out of them.\n",
    "\n",
    "###One-hot encoding to make dummy variables\n",
    "A problem with representing categorical variables as integers is that integers are ordered, while categories are not.  The standard way to deal with this is to use dummy variables; one-hot encoding is a very common way of dummying.  Each possible category becomes a new boolean feature.  For example, if our dataframe looked like this: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``index      country\n",
    "1          \"United States\"\n",
    "2          \"Mexico\"\n",
    "3          \"Mexico\"\n",
    "4          \"Canada\"\n",
    "5          \"United States\"\n",
    "6          \"Canada\"``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "then after dummying it will look something like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``index      country_UnitedStates     country_Mexico    country_Canada\n",
    "1          1                        0                 0\n",
    "2          0                        1                 0\n",
    "3          0                        1                 0\n",
    "4          0                        0                 1\n",
    "5          1                        0                 0\n",
    "6          0                        0                 1``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully the origin of the name is clear--each variable is now encoded over several boolean columns, one of which is true (hot) and the others are false.\n",
    "\n",
    "Now we'll write a hot-encoder function that takes the data frame and the title of a column, and returns the same data frame but one-hot encoding performed on the indicated feature.\n",
    "\n",
    "Protip: sklearn has a [one-hot encoder](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html) function available that will be your friend here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import sklearn.preprocessing\n",
    "\n",
    "def hot_encoder(df, column_name):\n",
    "    column = df[column_name].tolist()\n",
    "    column = np.reshape( column, (len(column), 1) )  ### needs to be an N x 1 numpy array\n",
    "    enc = sklearn.preprocessing.OneHotEncoder()\n",
    "    enc.fit( column )\n",
    "    new_column = enc.transform( column ).toarray()\n",
    "    column_titles = []\n",
    "    ### making titles for the new columns, and appending them to dataframe\n",
    "    for ii in range( len(new_column[0]) ):\n",
    "        this_column_name = column_name+\"_\"+str(ii)\n",
    "        df[this_column_name] = new_column[:,ii]\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll take the ``to_transform`` list that you populated above with categorical variables, and use that to loop through columns that will be one-hot encoded.\n",
    "\n",
    "One note before you code that up: one-hot encoding comes with the baggage that it makes your dataset bigger--sometimes a lot bigger.  In the countries example above, one column that encoded the country has now been expanded out to three columns.  You can imagine that this can sometimes get really, really big (imagine a column encoding all the counties in the United States, for example).  \n",
    "\n",
    "There are some columns in this example that will really blow up the dataset, so we'll remove them before proceeding with the one-hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['amount_tsh' 'funder' 'gps_height' 'installer' 'longitude' 'latitude'\n",
      " 'wpt_name' 'num_private' 'basin' 'subvillage' 'region' 'region_code'\n",
      " 'district_code' 'lga' 'ward' 'population' 'public_meeting' 'recorded_by'\n",
      " 'scheme_management' 'scheme_name' 'permit' 'construction_year'\n",
      " 'extraction_type' 'extraction_type_group' 'extraction_type_class'\n",
      " 'management' 'management_group' 'payment' 'payment_type' 'water_quality'\n",
      " 'quality_group' 'quantity' 'quantity_group' 'source' 'source_type'\n",
      " 'source_class' 'waterpoint_type' 'waterpoint_type_group']\n",
      "       amount_tsh  gps_height  longitude   latitude  num_private  basin  \\\n",
      "id                                                                        \n",
      "69572        6000        1390  34.938093  -9.856322            0      5   \n",
      "8776            0        1399  34.698766  -2.147466            0      8   \n",
      "34310          25         686  37.460664  -3.821329            0      0   \n",
      "67743           0         263  38.486161 -11.155298            0      7   \n",
      "19728           0           0  31.130847  -1.825359            0      8   \n",
      "\n",
      "       region  region_code  district_code  lga           ...             \\\n",
      "id                                                       ...              \n",
      "69572      13           11              5   75           ...              \n",
      "8776        1           20              2   76           ...              \n",
      "34310      17           21              4   41           ...              \n",
      "67743      19           90             63   82           ...              \n",
      "19728       2           18              1   48           ...              \n",
      "\n",
      "       waterpoint_type_3  waterpoint_type_4  waterpoint_type_5  \\\n",
      "id                                                               \n",
      "69572                  1                  0                  0   \n",
      "8776                   1                  0                  0   \n",
      "34310                  0                  0                  0   \n",
      "67743                  0                  0                  0   \n",
      "19728                  1                  0                  0   \n",
      "\n",
      "       waterpoint_type_6  waterpoint_type_group_0  waterpoint_type_group_1  \\\n",
      "id                                                                           \n",
      "69572                  0                        0                        0   \n",
      "8776                   0                        0                        0   \n",
      "34310                  0                        0                        0   \n",
      "67743                  0                        0                        0   \n",
      "19728                  0                        0                        0   \n",
      "\n",
      "       waterpoint_type_group_2  waterpoint_type_group_3  \\\n",
      "id                                                        \n",
      "69572                        1                        0   \n",
      "8776                         1                        0   \n",
      "34310                        1                        0   \n",
      "67743                        1                        0   \n",
      "19728                        1                        0   \n",
      "\n",
      "       waterpoint_type_group_4  waterpoint_type_group_5  \n",
      "id                                                       \n",
      "69572                        0                        0  \n",
      "8776                         0                        0  \n",
      "34310                        0                        0  \n",
      "67743                        0                        0  \n",
      "19728                        0                        0  \n",
      "\n",
      "[5 rows x 3031 columns]\n"
     ]
    }
   ],
   "source": [
    "print(features_df.columns.values)\n",
    "\n",
    "\n",
    "features_df.drop( \"funder\", axis=1, inplace=True )\n",
    "features_df.drop( \"installer\", axis=1, inplace=True )\n",
    "features_df.drop( \"wpt_name\", axis=1, inplace=True )\n",
    "features_df.drop( \"subvillage\", axis=1, inplace=True )\n",
    "features_df.drop( \"ward\", axis=1, inplace=True )\n",
    "\n",
    "names_of_columns_to_transform.remove(\"funder\")\n",
    "names_of_columns_to_transform.remove(\"installer\")\n",
    "names_of_columns_to_transform.remove(\"wpt_name\")\n",
    "names_of_columns_to_transform.remove(\"subvillage\")\n",
    "names_of_columns_to_transform.remove(\"ward\")\n",
    "\n",
    "for feature in names_of_columns_to_transform:\n",
    "    features_df = hot_encoder( features_df, feature )\n",
    "    \n",
    "print( features_df.head() )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the features are a little fixed up, I'd invite you to rerun the models, and see if the cross_val_score goes up as a result.  It is also a great chance to take some of the theory discussion from the workshop and play around with the parameters of your models, and see if you can increase their scores that way.  There's a blank code box below where you can play around."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.78060606  0.78308081  0.77585859]\n"
     ]
    }
   ],
   "source": [
    "X = features_df.as_matrix()\n",
    "y = labels_df[\"status_group\"].tolist()\n",
    "\n",
    "clf = sklearn.ensemble.RandomForestClassifier()\n",
    "score = sklearn.cross_validation.cross_val_score( clf, X, y )\n",
    "print(score)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "##End-to-end workflows using Pipeline and GridSearchCV\n",
    "\n",
    "So far we have made a nice workflow using a few ideas assembled in a script-like workflow.  A few spots remain where we can tighten things up though:\n",
    "\n",
    "* the best model, the random forest, has a lot of parameters that we'd have to work through if we really wanted to tune it\n",
    "* after dummying, we have _lots_ of features, probably only a subset of which are really offering any discriminatory power (this is a version of the bias-variance tradeoff)\n",
    "* maybe there's a way to make the code more streamlined (hint: there is)\n",
    "\n",
    "We will solve all these with two related and lovely tools in sklearn: Pipeline and GridSearchCV.\n",
    "\n",
    "Pipeline in sklearn is a tool for chaining together multiple pieces of a workflow into a single coherent analysis.  In our example, we will chain together a tool for feature selection, to will address the second point, which then feeds our optimized feature set into the random forest model, all in a few lines of code (which addresses the third point).\n",
    "\n",
    "To get to the first point, about finding the best parameters--that's where the magic of GridSearchCV comes in.  But first we need to get the feature selector and pipeline up and running, so let's do that now.\n",
    "\n",
    "In ``sklearn.feature_selection`` there is a useful tool, ``SelectKBest`` [(link)](http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectKBest.html) that you should use.  By default, this will select the 10 best features; that seems like it might be too few features to do well on this problem, so change the number of features to 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(59400, 100)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/civisemployee/.py34/lib/python3.4/site-packages/sklearn/feature_selection/univariate_selection.py:111: UserWarning: Features [ 12 191] are constant.\n",
      "  UserWarning)\n"
     ]
    }
   ],
   "source": [
    "import sklearn.feature_selection\n",
    "\n",
    "select = sklearn.feature_selection.SelectKBest(k=100)\n",
    "selected_X = select.fit_transform(X, y)\n",
    "\n",
    "print( selected_X.shape )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pipeline\n",
    "\n",
    "After selecting the 100 best features, the natural next step would be to run our random forest again to see if it does a little better with fewer features.  So we would have ``SelectKBest`` doing selection, with the output of that process going straight into a classifier.  A [Pipeline](http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) packages the transformation step of ``SelectKBest`` with the estimation step of ``RandomForestClassifier`` into a coherent workflow.\n",
    "\n",
    "Why might you want to use ``Pipeline`` instead of keeping the steps separate?\n",
    "\n",
    "  * makes code more readable\n",
    "  * don't have to worry about keeping track data during intermediate steps, for example between transforming and estimating\n",
    "  * makes it trivial to move ordering of the pipeline pieces, or to swap pieces in and out\n",
    "  * *Allows you to do GridSearchCV on your workflow*\n",
    "\n",
    "This last point is, in my opinion, the most important.  We will get to it very soon, but first let's get a pipeline up and running that does ``SelectKBest`` followed by ``RandomForestClassifier``.\n",
    "\n",
    "In the code box below, I've also set up a slightly better training/testing structure, where I am explicitly splitting the data into training and testing sets which we'll use below.  The  training/testing split before was handled automatically in ``cross_val_score,`` but we'll be using a different evaluation metric from here forward, the classification report, which requires us to handle the train/test split ourselves.\n",
    "\n",
    "Note: when you do ``SelectKBest``, you might see a warning about a bunch of features that are constant.  This isn't a problem.  It's giving you a heads up that the indicated features don't show any variation, which could be a signal that something is wrong or that ``SelectKBest`` might be doing something unexpected.  I"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.80      0.56      0.66      5007\n",
      "          1       0.12      0.00      0.01       942\n",
      "          2       0.69      0.92      0.79      7119\n",
      "\n",
      "avg / total       0.69      0.72      0.68     13068\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/civisemployee/.py34/lib/python3.4/site-packages/sklearn/feature_selection/univariate_selection.py:111: UserWarning: Features [  12  191  214  219  262  267  274  277  284  291  312  322  334  336  346\n",
      "  372  376  384  397  411  437  460  474  475  479  483  511  525  536  559\n",
      "  562  586  606  610  650  652  658  659  704  737  738  741  762  774  785\n",
      "  796  807  810  830  851  878  884  891  917  931  960  970  989 1003 1045\n",
      " 1057 1058 1060 1061 1083 1097 1130 1153 1154 1177 1178 1226 1228 1231 1241\n",
      " 1256 1261 1275 1281 1295 1302 1327 1330 1331 1392 1401 1403 1409 1422 1461\n",
      " 1498 1502 1581 1587 1602 1604 1627 1630 1645 1650 1655 1664 1680 1684 1689\n",
      " 1693 1702 1704 1714 1756 1784 1798 1802 1827 1833 1854 1909 1910 1939 1940\n",
      " 1968 2017 2087 2105 2120 2133 2138 2170 2271 2283 2318 2351 2360 2376 2380\n",
      " 2390 2397 2401 2406 2424 2429 2433 2464 2470 2477 2482 2483 2509 2523 2527\n",
      " 2528 2531 2535 2561 2588 2596 2601 2628 2630 2637 2669 2674 2708 2726 2737\n",
      " 2759 2771 2804 2815 2832 2852 2862 2889 2894 2900] are constant.\n",
      "  UserWarning)\n"
     ]
    }
   ],
   "source": [
    "import sklearn.pipeline\n",
    "\n",
    "select = sklearn.feature_selection.SelectKBest(k=100)\n",
    "clf = sklearn.ensemble.RandomForestClassifier()\n",
    "\n",
    "steps = [('feature_selection', select),\n",
    "         ('random_forest', clf)]\n",
    "\n",
    "pipeline = sklearn.pipeline.Pipeline(steps)\n",
    "\n",
    "X_train, X_test, y_train, y_test = sklearn.cross_validation.train_test_split(X, y, test_size=0.33, random_state=42)\n",
    "\n",
    "### fit your pipeline on X_train and y_train\n",
    "pipeline.fit( X_train, y_train )\n",
    "### call pipeline.predict() on your X_test data to make a set of test predictions\n",
    "y_prediction = pipeline.predict( X_test )\n",
    "### test your predictions using sklearn.classification_report()\n",
    "report = sklearn.metrics.classification_report( y_test, y_prediction )\n",
    "### and print the report\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading the classification report\n",
    "\n",
    "A brief aside--we've switched from ``cross_val_score`` to ``classification_report`` for evaluation, mostly to show you two different ways to evaluating a model.  The classification report has the advantage of giving you a lot more information, and if (for example) one class is more important to get right than the others (say you're trying to zero in on non-functional wells, so finding those correctly is more important than getting the functional wells right).\n",
    "\n",
    "For more information, the [sklearn docs](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) on ``classification_report`` are, like all the sklearn docs, incredibly helpful.  For interpreting the various metrics, [this page](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html) may also help.\n",
    "\n",
    "###GridSearchCV\n",
    "We're in the home stretch now. When we decided to select the 100 best features, setting that number to 100 was kind of a hand-wavey decision. Similarly, the RandomForestClassifier that we're using right now has all its parameters set to their default values, which might not be optimal.\n",
    "\n",
    "So, a straightforward thing to do now is to try different values of ``k`` and any ``RandomForestClassifier`` parameters we want to tune (for the sake of concreteness, let's play with n_estimators and min_samples_split). Trying lots of values for each of these free parameters is tedious, and there can sometimes be interactions between the choices you make in one step and the optimal value for a downstream step. In other words, to avoid local optima, you should try all the combinations of parameters, and not just vary them independently. So if you want to try 5 different values each for ``k``, ``n_estimators`` and ``min_samples_split``, that means 5 x 5 x 5 = 125 different combinations to try. Not something you want to do by hand.\n",
    "\n",
    "``GridSearchCV`` allows you to construct a grid of all the combinations of parameters, tries each combination, and then reports back the best combination/model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'random_forest': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
      "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
      "            min_samples_leaf=1, min_samples_split=2,\n",
      "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=1,\n",
      "            oob_score=False, random_state=None, verbose=0,\n",
      "            warm_start=False), 'feature_selection': SelectKBest(k=20, score_func=<function f_classif at 0x1119a4598>)}\n",
      "[0 2 2 2 2 0 2 0 0 2 2 2 0 2 2 2 2 2 0 0]\n"
     ]
    }
   ],
   "source": [
    "import sklearn.grid_search\n",
    "#import warnings\n",
    "#warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "\n",
    "parameters = dict(feature_selection__k=[100, 200], \n",
    "              random_forest__n_estimators=[50],\n",
    "              random_forest__min_samples_split=[4])\n",
    "\n",
    "cv = sklearn.grid_search.GridSearchCV(pipeline, param_grid=parameters)\n",
    "print(pipeline.named_steps)\n",
    "cv.fit(X_train, y_train)\n",
    "y_predictions = cv.predict(X_test)\n",
    "report = sklearn.metrics.classification_report( y_test, y_predictions )\n",
    "### and print the report\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}