WestNileVirus_Prediction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting outbreaks of West Nile Virus\n",
    "\n",
    "Qiaolin Chen, Ph.D. Aug 2017\n",
    "\n",
    "### Goal\n",
    "Here I analyze weather data and GIS data and predict whether or not West Nile virus is present, for a given time, location, and species. \n",
    "\n",
    "The analysis aim to answer the following questions:\n",
    " \n",
    "1. How predictive modeling could be used to inform actions that the City of Chicago might take and suggest any preventative actions.\n",
    "* To understand the factors that are related to / predictive of the spread of WNV, what modeling choices / evaluation metrics would be most appropriate?\n",
    "* What additional data might be relevant for understanding this problem?\n",
    "* Are there any lessons for future data collection efforts?\n",
    "* What weather conditions would we expect to impact mosquito populations? How would you investigate this using this data?\n",
    "\n",
    "\n",
    "### Problem Description \n",
    "(from https://www.kaggle.com/c/predict-west-nile-virus)\n",
    "West Nile virus (WNV) is most commonly spread to humans through infected mosquitoes. Around 20% of people who become infected with the virus develop symptoms ranging from a persistent fever, to serious neurological illnesses that can result in death.\n",
    "\n",
    "In 2002, the first human cases of WNV were reported in Chicago. By 2004 the City of Chicago and the Chicago Department of Public Health (CDPH) had established a comprehensive surveillance and control program that is still in effect today.\n",
    "\n",
    "Every week from late spring through the fall, mosquitoes in traps across the city are tested for the virus. The results of these tests influence when and where the city will spray airborne pesticides to control adult mosquito populations.\n",
    "\n",
    "**Given weather, location, testing, and spraying data, this analysis predict when and where different species of mosquitos will test positive for WNV. A more accurate method of predicting outbreaks of WNV in mosquitoes will help the City of Chicago and CDPH more efficiently and effectively allocate resources towards preventing transmission of this potentially deadly virus. **\n",
    "\n",
    "\n",
    "### Data\n",
    "The relevant data for this challenge can be downloaded here: https://www.kaggle.com/c/predict-west-nile-virus/data\n",
    "\n",
    "1. **Main dataset**:\n",
    "Train and test data include record ID, date that the WNV test is performed, the species of mosquitos, Id of the trap, number of mosquitoes caught in this trap, and trap location variables (approximate address, block number, street name, approximate address, latitude and longitude).\n",
    "The target label is whether West Nile Virus was present in these mosquitos (1= WNV is present,0= not present). \n",
    "\n",
    "2. **Spray Data**:\n",
    "The City of Chicago also does spraying to kill mosquitos. You are given the GIS data for their spray efforts in 2011 and 2013. Spraying can reduce the number of mosquitos in the area, and therefore might eliminate the appearance of West Nile virus. \n",
    "\n",
    "2. **Weather Data**:\n",
    "It is believed that hot and dry conditions are more favorable for West Nile virus than cold and wet. The weather data come from NOAA of the weather conditions of 2007 to 2014, during the months of the tests. \n",
    "\n",
    "2. **Map Data**:\n",
    "The map files mapdata_copyright_openstreetmap_contributors.rds and mapdata_copyright_openstreetmap_contributors.txt are from Open Streetmap and are primarily provided for use in visualizations.\n",
    "\n",
    "###  Steps\n",
    "\n",
    "1. Clean the dataset \n",
    "2. Engineer features \n",
    "3. Predictive models \n",
    "4. Present results\n",
    " \n",
    "Here I explore the data, construct and/or select features, and use those features in one or more predictive models. I follow an iterative approach, establishing an initial baseline of performance upon which subsequent efforts can build.\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import necessary packages\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn\n",
    "from datetime import datetime\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "from sklearn import ensemble, preprocessing\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import base\n",
    "from sklearn.preprocessing import OneHotEncoder, LabelEncoder\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV \n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import accuracy_score, precision_score, recall_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# read data as a pandas dataframe\n",
    "data_path = \"/Users/qiaolinchen/Documents/data_challenge/sparkbeyond/data/\"\n",
    "train   = pd.read_csv(data_path + 'train.csv')\n",
    "test    = pd.read_csv(data_path + 'test.csv')\n",
    "weather = pd.read_csv(data_path + 'weather.csv')\n",
    "spray = pd.read_csv(data_path + 'spray.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Feature Generation and Exploratory Data Analysis\n",
    "### 1.1 Read the CSV data file as pandas DataFrame\n",
    "First we load and explore all datasets. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Main dataset\n",
    "\n",
    "\n",
    "Please note that not all the locations are tested at all times. Also, records exist only when a particular species of mosquitos is found at a certain trap at a certain time. In the test set, we ask you for all combinations/permutations of possible predictions and are only scoring the observed ones.\n",
    "* The training set consists of data from 2007, 2009, 2011, and 2013.\n",
    "* The test set includes the test results for 2008, 2010, 2012, and 2014."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label: WnvPresent\n",
    "Whether West Nile Virus was present (WnvPresent) is a binary variable to predict (1= present, 0= absent). \n",
    "- This is a classification problem. \n",
    "- The mean is 0.0524, suggest if we using a simplest bench mark model,predicting absent of WNV for all future cases, the accuracy will be 95%. \n",
    "- Therefore, **accuracy is not a good metric** to access performance of classifiers.\n",
    "- **Precision, recall and F-1 score** will be more appropriate metrics. \n",
    "- Alternatively, **area under the curve (AUC)of ROC** is a good metric assessing overall performance of classifiers regardless of cutpoint. \n",
    "\n",
    "Other variables: \n",
    "- **Date**: date that the WNV test is performed. \n",
    "    - Will extract year and month from Date. Date will also be used to merge with weather data. \n",
    "- **NumMosquitos**: number of mosquitoes caught in this trap. capped at 50. If exceed 50, split into another row.\n",
    "    - Need to merge these split records back to get the correct NumMosquitos.  \n",
    "- **Species**: the species of mosquitos. Combine 7 species into 4. \n",
    "    - From count by species below, will combine 4 minor species CULEX ERRATICUS, CULEX SALINARIUS, CULEX TARSALIS and CULEX TERRITANS in to a group OTHER. \n",
    "\n",
    "- **Location** related features: will keep **Street, Latitude, Longitude**\n",
    "   - Address (drop): approximate address of the location of trap, 138 unique address. \n",
    "   - AddressNumberAndStreet (drop): approximate address returned from GeoCoder, 138 unique address.\n",
    "   - Block(drop): block number of address, 64 unique blocks, suggesting different street can have the same block number. This variable depending on East/West/North/South of Street is linearly related to latittude and longitude. So I drop this variable. \n",
    "   - **Street**: street name, 138 streets. There are 136 traps on 138 streets, so street and traps have also 1-1 correpondence. Will only use street in this project. \n",
    "   - Trap (drop): Id of the trap. There are 136 unique Trap. only 2 Satellite traps T094B and T054C in training set.\n",
    "   - **Latitude, Longitude**: Latitude and Longitude returned from GeoCoder\n",
    "   - AddressAccuracy(drop): accuracy returned from GeoCoder\n",
    "   \n",
    "The location of the traps are described by the block number and street name. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Address</th>\n",
       "      <th>Species</th>\n",
       "      <th>Block</th>\n",
       "      <th>Street</th>\n",
       "      <th>Trap</th>\n",
       "      <th>AddressNumberAndStreet</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>AddressAccuracy</th>\n",
       "      <th>NumMosquitos</th>\n",
       "      <th>WnvPresent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2007-05-29</td>\n",
       "      <td>4100 North Oak Park Avenue, Chicago, IL 60634,...</td>\n",
       "      <td>CULEX PIPIENS/RESTUANS</td>\n",
       "      <td>41</td>\n",
       "      <td>N OAK PARK AVE</td>\n",
       "      <td>T002</td>\n",
       "      <td>4100  N OAK PARK AVE, Chicago, IL</td>\n",
       "      <td>41.95469</td>\n",
       "      <td>-87.800991</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2007-05-29</td>\n",
       "      <td>4100 North Oak Park Avenue, Chicago, IL 60634,...</td>\n",
       "      <td>CULEX RESTUANS</td>\n",
       "      <td>41</td>\n",
       "      <td>N OAK PARK AVE</td>\n",
       "      <td>T002</td>\n",
       "      <td>4100  N OAK PARK AVE, Chicago, IL</td>\n",
       "      <td>41.95469</td>\n",
       "      <td>-87.800991</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date                                            Address  \\\n",
       "0  2007-05-29  4100 North Oak Park Avenue, Chicago, IL 60634,...   \n",
       "1  2007-05-29  4100 North Oak Park Avenue, Chicago, IL 60634,...   \n",
       "\n",
       "                  Species  Block           Street  Trap  \\\n",
       "0  CULEX PIPIENS/RESTUANS     41   N OAK PARK AVE  T002   \n",
       "1          CULEX RESTUANS     41   N OAK PARK AVE  T002   \n",
       "\n",
       "              AddressNumberAndStreet  Latitude  Longitude  AddressAccuracy  \\\n",
       "0  4100  N OAK PARK AVE, Chicago, IL  41.95469 -87.800991                9   \n",
       "1  4100  N OAK PARK AVE, Chicago, IL  41.95469 -87.800991                9   \n",
       "\n",
       "   NumMosquitos  WnvPresent  \n",
       "0             1           0  \n",
       "1             1           0  "
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#train = train.drop(['AddressNumberAndStreet'], axis=1)\n",
    "train.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summarize numeric variables: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>AddressAccuracy</th>\n",
       "      <th>NumMosquitos</th>\n",
       "      <th>WnvPresent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10506.000000</td>\n",
       "      <td>10506.000000</td>\n",
       "      <td>10506.000000</td>\n",
       "      <td>10506.000000</td>\n",
       "      <td>10506.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>41.841139</td>\n",
       "      <td>-87.699908</td>\n",
       "      <td>7.819532</td>\n",
       "      <td>12.853512</td>\n",
       "      <td>0.052446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.112742</td>\n",
       "      <td>0.096514</td>\n",
       "      <td>1.452921</td>\n",
       "      <td>16.133816</td>\n",
       "      <td>0.222936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>41.644612</td>\n",
       "      <td>-87.930995</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>41.732984</td>\n",
       "      <td>-87.760070</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>41.846283</td>\n",
       "      <td>-87.694991</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>41.954690</td>\n",
       "      <td>-87.627796</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>42.017430</td>\n",
       "      <td>-87.531635</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Latitude     Longitude  AddressAccuracy  NumMosquitos    WnvPresent\n",
       "count  10506.000000  10506.000000     10506.000000  10506.000000  10506.000000\n",
       "mean      41.841139    -87.699908         7.819532     12.853512      0.052446\n",
       "std        0.112742      0.096514         1.452921     16.133816      0.222936\n",
       "min       41.644612    -87.930995         3.000000      1.000000      0.000000\n",
       "25%       41.732984    -87.760070         8.000000      2.000000      0.000000\n",
       "50%       41.846283    -87.694991         8.000000      5.000000      0.000000\n",
       "75%       41.954690    -87.627796         9.000000     17.000000      0.000000\n",
       "max       42.017430    -87.531635         9.000000     50.000000      1.000000"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[['Species', 'Trap', 'Latitude', 'Longitude', 'AddressAccuracy',\n",
    "       'NumMosquitos', 'WnvPresent']].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summarize categorical variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Address</th>\n",
       "      <th>AddressNumberAndStreet</th>\n",
       "      <th>Species</th>\n",
       "      <th>Street</th>\n",
       "      <th>Trap</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>10506</td>\n",
       "      <td>10506</td>\n",
       "      <td>10506</td>\n",
       "      <td>10506</td>\n",
       "      <td>10506</td>\n",
       "      <td>10506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>95</td>\n",
       "      <td>138</td>\n",
       "      <td>138</td>\n",
       "      <td>7</td>\n",
       "      <td>128</td>\n",
       "      <td>136</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>2007-08-01</td>\n",
       "      <td>ORD Terminal 5, O'Hare International Airport, ...</td>\n",
       "      <td>1000  W OHARE AIRPORT, Chicago, IL</td>\n",
       "      <td>CULEX PIPIENS/RESTUANS</td>\n",
       "      <td>W OHARE AIRPORT</td>\n",
       "      <td>T900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>551</td>\n",
       "      <td>750</td>\n",
       "      <td>750</td>\n",
       "      <td>4752</td>\n",
       "      <td>750</td>\n",
       "      <td>750</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              Date                                            Address  \\\n",
       "count        10506                                              10506   \n",
       "unique          95                                                138   \n",
       "top     2007-08-01  ORD Terminal 5, O'Hare International Airport, ...   \n",
       "freq           551                                                750   \n",
       "\n",
       "                    AddressNumberAndStreet                 Species  \\\n",
       "count                                10506                   10506   \n",
       "unique                                 138                       7   \n",
       "top     1000  W OHARE AIRPORT, Chicago, IL  CULEX PIPIENS/RESTUANS   \n",
       "freq                                   750                    4752   \n",
       "\n",
       "                  Street   Trap  \n",
       "count              10506  10506  \n",
       "unique               128    136  \n",
       "top      W OHARE AIRPORT   T900  \n",
       "freq                 750    750  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[['Date', 'Address', 'AddressNumberAndStreet', 'Species', 'Street', 'Trap']].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Species: \n",
    "CULEX PIPIENS/RESTUANS(N=66268, p= 0.055135 ), CULEX PIPIENS (N=44671, p= 0.088922), CULEX RESTUANS(N= 23431, )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">NumMosquitos</th>\n",
       "      <th colspan=\"3\" halign=\"left\">WnvPresent</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Species</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CULEX ERRATICUS</th>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CULEX PIPIENS</th>\n",
       "      <td>2699</td>\n",
       "      <td>44671</td>\n",
       "      <td>16.550945</td>\n",
       "      <td>2699</td>\n",
       "      <td>240</td>\n",
       "      <td>0.088922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CULEX PIPIENS/RESTUANS</th>\n",
       "      <td>4752</td>\n",
       "      <td>66268</td>\n",
       "      <td>13.945286</td>\n",
       "      <td>4752</td>\n",
       "      <td>262</td>\n",
       "      <td>0.055135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CULEX RESTUANS</th>\n",
       "      <td>2740</td>\n",
       "      <td>23431</td>\n",
       "      <td>8.551460</td>\n",
       "      <td>2740</td>\n",
       "      <td>49</td>\n",
       "      <td>0.017883</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CULEX SALINARIUS</th>\n",
       "      <td>86</td>\n",
       "      <td>145</td>\n",
       "      <td>1.686047</td>\n",
       "      <td>86</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CULEX TARSALIS</th>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>1.166667</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CULEX TERRITANS</th>\n",
       "      <td>222</td>\n",
       "      <td>510</td>\n",
       "      <td>2.297297</td>\n",
       "      <td>222</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       NumMosquitos                   WnvPresent       \\\n",
       "                              count    sum       mean      count  sum   \n",
       "Species                                                                 \n",
       "CULEX ERRATICUS                   1      7   7.000000          1    0   \n",
       "CULEX PIPIENS                  2699  44671  16.550945       2699  240   \n",
       "CULEX PIPIENS/RESTUANS         4752  66268  13.945286       4752  262   \n",
       "CULEX RESTUANS                 2740  23431   8.551460       2740   49   \n",
       "CULEX SALINARIUS                 86    145   1.686047         86    0   \n",
       "CULEX TARSALIS                    6      7   1.166667          6    0   \n",
       "CULEX TERRITANS                 222    510   2.297297        222    0   \n",
       "\n",
       "                                  \n",
       "                            mean  \n",
       "Species                           \n",
       "CULEX ERRATICUS         0.000000  \n",
       "CULEX PIPIENS           0.088922  \n",
       "CULEX PIPIENS/RESTUANS  0.055135  \n",
       "CULEX RESTUANS          0.017883  \n",
       "CULEX SALINARIUS        0.000000  \n",
       "CULEX TARSALIS          0.000000  \n",
       "CULEX TERRITANS         0.000000  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train[['NumMosquitos','WnvPresent', 'Species']].groupby('Species').agg(['count', 'sum','mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "train   = pd.read_csv(data_path + 'train.csv')\n",
    "\n",
    "def get_date(x):\n",
    "    return datetime.strptime(x , '%Y-%m-%d').date()\n",
    "\n",
    "# recode species, group small species to OTHER\n",
    "def group_species(x):\n",
    "    if x in ['CULEX ERRATICUS', 'CULEX SALINARIUS', 'CULEX TARSALIS', 'CULEX TERRITANS' ]:\n",
    "        return 'OTHER'\n",
    "    else :  return x\n",
    "    \n",
    "train['Species2']  =list( map( group_species , train['Species']))\n",
    "# get year and month\n",
    "train['Date'] = list(map(get_date   , train['Date']))\n",
    "train['Year'] = [ x.year for x in train['Date']]\n",
    "train['Month']  = [x.month for x in train['Date']]\n",
    "train['Week']  = [x.strftime(\"%V\") for x in train['Date']]\n",
    "# get E, W , S,N from street\n",
    "train['EWSN'] = [x.split()[0] for x in train['Street'] ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1143f2588>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuEAAAEnCAYAAAAHGZ2mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucXGWd7/vPLx2gEUkQSOTShLBlgDgEI/RWSGYOMWGU\nTQID+wgJI2NAQ3QrkoCAZMatEeEQBQ0wnq2TDRIYGBIQhOEmYDReSMCdQKTRcPFoYHLBJFyCIIkk\n+Z0/1qqmUl1VXdVV6/59v14FVWvVWv1UV3rVt571e55l7o6IiIiIiMRnUNINEBEREREpGoVwERER\nEZGYKYSLiIiIiMRMIVxEREREJGYK4SIiIiIiMVMIFxERERGJmUK4iIjUZGZjzOwxM1tpZsvN7EPh\n8ovDZSvN7Gkz225me1fZfoGZ/aHsuWPifxUiIuljmidcRERqMbOHgXnu/qCZnQRc4u7jK55zMnCB\nu0+osv0C4D53/0Ec7RURyQr1hIuISD0ODAnvDwXWVXnOmcBtsbVIRCQHCtETvu+++/rIkSOTboaI\nSNNWrFixyd2HJfXzzWwU8BBgBB03Y939hbL17wLWAIe6+ytVtl8AHAdsBRYDl7r71irPmwHMANhj\njz2OOeKII9r/YkREItbMMbsQIby7u9uXL1+edDNERJpmZivcvTvin/FjYL8qq/4ZmAj8zN3vNLMz\ngBnufkLZtlOAs9z95Br73h94CdgVmA/8f+5+Wb326JgtIlnVzDF7cNSNERGRdCsP1ZXM7GZgZvjw\nDuD6iqdMpU4piruvD+9uNbMbgYtaaKqISG6oJlxEROpZBxwf3p8APF9aYWZDw3X31No47AnHzAw4\nFXg6spaKiGSIesJFRKSec4FrzWwwsIWwbjt0GvCwu79ZvoGZPQBMd/d1wK1mNoygpnwl8Nl4mi0i\nkm4K4SIiUpO7/xI4psa6BcCCKstPKrvfZ9pCERFROYqIiIiISOwUwkVEREREYqYQLiIiIiISM4Vw\nEREREZGYaWCmiAzcnKFVlm2Ovx0iIiIZo55wERmYagG83nIRERHppRAuIiIiIhIzlaOISHUqNRER\nEYmMQvgArTpiVJ9lo55ZlUBLRNqov1KSOUMVxEVERNpAIXwAqgXw0nIFcckc1XCLSD9GXnp/n2Wr\n505KoCUi0Rh90+g+y3qm9UT6MxXCE6YedUlUKwF8zmaVrIjkWLXgXbleQVzyoFoALy2PMogrhCdI\nPeqSeQrckmJJ9GzlRX8BXERapxAuIiK5k1TPVtGoTEVk4DRFoYg0Tj3fItIP9aKLNCbyEG5mHWb2\npJndFz6+1cyeNbOnzez7ZrZLje2mmdnz4W1a2fJjzKzHzH5nZteZmUX9GirVKhVRCYnk0pzN79xE\nRESkLeIoR5kJrAKGhI9vBc4K7/87MB34bvkGZrY38FWgG3BghZn9h7u/Gj73XOBx4AHgRODBiF9D\nHwrckhqtDI7U4EopINWKi0i5nmk9+Zsdxcy6gEnAFcCFAO7+QNn6XwFdVTb9GPCIu78SPu8R4EQz\nWwIMcffHwuU3A6eSQAhvh1HPrNLsKNKaepeObyaIixScasV3tnrupH7rvVV2InmSxN9/1D3h1wCX\nAHtWrgjLUP6RoKe80oHAf5Y9XhMuOzC8X7m8DzObAcwAGDFixACa3rpGArYCt9SlXmqRAanVsyWN\n0wBLkWhFFsLNbDKwwd1XmNn4Kk/5X8DP3f0XUfx8d58PzAfo7u72KH5GPZp+UFrWjl5u6ZfORuVX\ntZ4tBfP2aaS3XERqi7InfBxwipmdBHQCQ8zsFnc/y8y+CgwDPlNj27XA+LLHXcCScHlXxfK1bW63\nSLJ0BcvY6MuySGsUuEUGLrIQ7u6zgdkAYU/4RWEAn05Q8z3R3XfU2Pwh4P8xs/eEjz8KzHb3V8zs\ndTM7lmBg5ieBf4nqNYjERsFbRESkUJK4WM/3gBeAZeHsgne5+2Vm1g181t2nh2H768D/Cbe5rDRI\nE/gcsADYnWBAZiYHZUpBtTtsa3YTERHJoSLMYhRLCHf3JQTlJLh71Z/p7ssJpissPf4+8P0azzsy\ninaKRCqq3m4FbhERyZGiXPFWV8yMiC7oI5FR6BYZsFof4Hn6YBeRbEiiHKUwFLilrRS+205z9RdP\nUXrYRCT9FMJFskABPBKaHUVEsqwIddN5phAuhZbqXlAFbxERqUFndbJPIVwKK1W9oArcIoWjC92I\nVFfrird5+3KhEC4iIhKzagG8tFxBXCR/gbsahfB+pLpcQaITxdzbmtNbf0+SuKL0sIlI+imE15Gq\ncgWJkVdfPGdoe4J4QaXx70mzoxSTAreIpIFCuEgfNUJ4iig4to9+byKSRTqrk30K4VJYNXtBp76U\nQGsal8YeZRERiZ8Cd7YphCdMPZrJqvq7jury8iIiodVzJ2l2FJGCUwhPkHo0RUSKS4FbpNgUwuvQ\noK0M6q8Xu5GBkZrFJBL6exIREXmHQng/FBAypJEykooZTmqGwhqBWyGyNfpdiYjkg8qpWjco6QaI\nJKVeOVA7ljfblspbLbWCrAKuiIjEod7FpqRx6gnvh3o+UyLH5SEDGRugf4Mi1WnKtnzI2/uYt9cj\n7aGe8Dqi7PkE9Wg2rFaZiWYxEZEy1YJOveWSTnl7H/P2eqR91BOeMAXuOhSyRUREElX9y8Lc2NuR\nR+oJl3SZM/SdW9PbNjjziYiIiPSrVm/9HodfGnNL8kk94ZK8dvZ4NxGym50yT1PsiYiIwKAaXbia\nHaU5kYdwM+sAlgNr3X2ymZ0HzALeBwxz9001tvsGUHo3v+7ui8LlC4DjgVLaOtvdV0b4EiQKKSk1\naTZARxG4Fe5FRCRrFLhbF0dP+ExgFTAkfPwocB+wpNYGZjYJOBoYA+wGLDGzB9399fApF7v7DyJr\ncUjhqM3aHbxzVFqif1MireuZ1qNZKHIgb+9j3l6PtE+kIdzMugh6s68ALgRw9yfDdfU2fT/wc3ff\nBmwzs6eAE4Hbo2xvNVGEo0IG+4RKTkSkNWY2Bvge0AlsAz7n7r8ys4uBT4RPGwyMIji7+UrF9gZc\nDpwObAe+6+7XRdXeWsEmjhCki5e0T94Cat5ej7RH1D3h1wCXAHs2ud2vga+a2beAdwEfAX5btv4K\nM/sKsBi41N23Vu7AzGYAMwBGjBgxgKZHYyBzQmeaer9Fsu6bwNfc/UEzOyl8PN7drwKuAjCzk4EL\nKgN46GzgIOAId99hZsOjbGy1ILznqOqDyEbfNLpt4ajexUsUxKVclr6sqRc/WpGFcDObDGxw9xVm\nNr6Zbd39YTP7r8BSYCOwjKAHBWA28BKwKzAf+BJwWZV9zA/X093d7QN8GdKKdgVwBW+RJDnvlBMO\nBdZVec6ZwG01tv8fwD+4+w4Ad9/Q9haGagVhd6h/8lUkHln8sqbAHZ0oe8LHAaeEPSedwBAzu8Xd\nz2pkY3e/gqCMBTP7d+C5cPn68ClbzexG4KK2t1ySV7DgXcgSJcmKWcBDZnY1wbS2Y8tXmtm7CMoF\nz6ux/fuAKWZ2GkGnyvnu/nzlk9p19nKPwy/dKXC7umBEIpel3v00iSyEu/tsgl5rwp7wixoN4OGM\nKnu5+8tmdhRwFPBwuG5/d18f1hmeCjwdRfulAe2+lHzBgndJ4UqUJHXM7MfAflVW/TMwkaDU5E4z\nOwO4ATih7DknA4/WKEWBYHD9FnfvNrP/Dnwf+NvKJ7Xj7GUpgKvXWyQ+WezdT4vY5wk3s/MJ6sT3\nA54yswfcfbqZdQOfdffpwC7AL8LBm68DZ4WDNAFuNbNhgAErgc/G/RqE+peSbyZMFzR4i6SJu59Q\na52Z3UwwyxXAHcD1FU+ZSu1SFIA1wF3h/R8CNw6wmf2qFsDN1BsuIukUSwh39yWEUxKGo+L7jIx3\n9+XA9PD+FoIZUqrta0JU7YyDpj0MKXyLZMU6gmszLAEmAL2lJGY2NFxX7yzn3QSD6/8QPve5qBra\nrHbWuq6eO0mn5EWkKbpiZgIKE7jnbG5/yYqIxO1c4FozGwxsIazbDp0GPOzub5ZvYGYPANPdfR0w\nl+AM5gXAG4SdLVGoVYZiFs/gMgXuFErZZ1Bav6xpBpRkKIQXWRwHJwVukUxz918Cx9RYtwBYUGX5\nSWX3X+Odqx+LxKddZZNtlnTgrlQtgJeWZy2IZ+3LxKCkGyAJqXdwkljVOjNSmDMmIm1S68M2zR/C\nkpyJiyYy+qbRvbeJiyYm3aRMqvWlIu4vG/W+TKSVesJlYFRq0lYK3CLtocAtjZi4aCIbtuw8Zf2G\nLRuYuGgii6csTqhV2ZW23v2sUAiXgVPgFhGRDKoM4P0tF4mCQnhWzHkPsKNswSCY82pSrck0zU4j\nIiIiSVNNeBb0CeAEj+e8J4nWZFq9C+OkxaojRvW5iYhIk2qdrdVZ3J1oLEVy1BMeofb1uFYG8LLl\npbrsZg8qqulOJV09U0SkjWp8pg3vHF619GR45/CoW5RKeQjcPdN6Mjc7ikJ4RGIPUwOZckmBuzBU\ngiNFlLUP5Cyq9ztO83Fn8ZTFfQZnDu8crkGZGZe1v2+FcJGcU++6FFGe5j5Oq3q/49uv3FZ1XZqO\nOwrckjSFcJGCS3NvlYjkjQ+8jFIkZxTCpVBGPbNKobOMeslFYqDxN9UlfOVKkaQphEvhpDlc6kuC\nSE5UndWqfL0CqEjRKYRHpG6YiqJXRAfz3FDgFsm4/gJ4YXl4ExHQPOGRGvXMqj63qgEcai/vz5zN\nCuBSV61Qr7AveZbs3MfFCOD1fsc7H1+89zZq6ktxNE0kE9QTHqWBBus++9Gc3tIaBW4pog462M72\nnR5Le9X7UtN73GnXZ6FIziiEt1tUBxsFbomAatAlr8bcNGanAA6wne2MuWkMK6etTKhVBaWOJJGq\nFMLbQd/yJcMUuCWPKgN4f8vbaxD9lqQULYBG8Hp1MSbJOoXwJlT9g//Diwm0REREUmvOq1UGZw4K\nlktb6GJMkgcK4Q2q+Qd/cBc9L6xp/QcUrVdERCQCpfKqReFjB6bOTuCjToFbRPoR+ZHJzDqA5cBa\nd59sZucBs4D3AcPcfVON7b4BTAofft3dF4XLDwEWAvsAK4B/dPe/RPYCSqUmIw8Cs2oNHcA+FbhF\nRNpt1RGjcMDCW8nCK7f1BnENzhTJhiKUG8UxReFMoLzo9FHgBOCFWhuY2STgaGAM8GHgIjMbEq7+\nBjDP3Q8FXgU+HUWjgWhqvRXARUQiUQrg5SoDeTw14SLSinrlRnkSaU+4mXUR9GZfAVwI4O5Phuvq\nbfp+4Ofuvg3YZmZPASea2R3ABOAfwufdBMwBvhtF+ystvHLbTgdzB6Ze2kCvioK3iIiIiJSJuhzl\nGuASYM8mt/s18FUz+xbwLuAjwG8JSlBeC8M5wBrgwGo7MLMZwAyAESNGNN/yCgvnbu/To1JaztQq\nGyh4i4ikxqIrtyVXH94ATRXanJ5pPZktV6icPrODDk2b2Y/yTtBVVwZ/K3n4+4jsaGRmk4EN7r7C\nzMY3s627P2xm/xVYCmwElkFz5xDdfT4wH6C7u7vl6+QOqnKiszeUK3CLiCSudKCvVpJSsvDKbTAt\npgY1qLyWvcTD5XkIGlHJQuCupPnrm1cK4JV/13n4+4iyS2AccIqZnQR0AkPM7BZ3P6uRjd39CoIy\nFszs34HngJeBvcxscNgb3gWsjaT1GadelYjoghMiqTV19uAgZIfqhfE0qVXL3nLvkaROsvPXZ1O1\nAJ4XkQ3MdPfZ7t7l7iMJCjZ+0mgAN7MOM9snvH8UcBTwsLs78FPg4+FTpwH3tL3xJRkNV9UCeL3l\n0qBaA3V1sSaRVOiZ1sPU2YOZUqfkJK8f5iJ5ksWzHAMRx+woOzGz881sDUEv9lNmdn24vLt0H9gF\n+IWZ/ZagpOSssjrwLwEXmtnvCGrEb4i0wXM2h2Fch24RkbTrmdZDz7SeukfsVUeMUqeESMo18rec\ndbGMUHH3JcCS8P51wHVVnrMcmB7e30IwQ0q1ff0e+FBETa1p1DOrVOIhIpIjaakprVbL7qgcJY86\n6KhaeqL564spncPEUyoNB2sREelfrY6Temp1tETdAVNZyw7vXOkzjSflszoryYC0eRzQymkrNTtK\nk/LcCaoQLgM28tL7+yxbPXdSlWeKiMTvjLA2fFHFNR6qqT+Wpu/cJe3sRe+Z1sNoshFs611EJY3t\nbUm9cUAtBvE8iDMY5yFwV6MQnkNxfGusFsBLy3MbxOds1uwoIhnRvivrxTN3Se4CrORavS+taQ/M\nExdNZMOWDb2Ph3cOZ/GUxYm0RSE8p9L+R5BZCtwimVNr/nARKZbKAA6wYcsGJi6amEgQVwiX7FFv\ntIg0odb84eqsECmWygDe3/KoKYRLtkRUoyci+VZ+ufp2lX6Un5JXoBeRZimEi4iING3n4pZ6tbBx\nDGCLa8aSnmk9xZkdReOAJGIK4TIgq+dO0uwoIpJazYbF2lMatlZJHscAtrhnLMll4K5FgbuqPE8b\nGCeFcBkwBW4RSbNmw2K1AKEra4pUp8DdOoXwfvQ3zVWhegRERApmIBf9ERFphEJ4HZUBfGHFBR8c\nGE0OL1CQZqrRE5GYlff4KZCLSLsohDeoFMArqwMXXrkNpiXRogJT4BaRhMRdC1uYQZAiMUjbwGKF\n8AZVC+C68IOISHpF9WEbVy1sowMu0xYsJMV0JjlVfxcK4SIikjtxzxhSS1w952kKFrmSp9Cq62yk\njkK4iEiOmNnBwF+5+4/NbHdgsLv/Kel2FZlmkagjzSFXoVUiphBeR/kpPg+XVQ7M9MqNREQSYmbn\nAjOAvYH3AV3A94CJSbarkNIcLtNCIVcKTiG8H6VTfKMZHQzCLOMEl0LO20lATcAvklmfBz4EPA7g\n7s+b2fBkmxSv/qaVjUUM4XL1Oefw1rLH6j5Hx22RdGs4hJvZ3xCc4rzRzIYB73b3P0TXtPSZOjv/\n31niuLqbiERmq7v/xSw4Z2dmgynQCbtUBPA2qjXg8t4l3f0GcNBxWyTtGkqVZvZVoBs4HLgR2AW4\nBRgXXdOk3OWPXc4dz93BDt/BIBvE6YedzpeP/XLSzRKRdPmZmf0TsLuZ/R3wOeDeVnZoZmMISlo6\ngW3A59z9V2Z2MfCJ8GmDgVHAMHd/pWL7XwB7hg+HA79y91NbaVORVBtwuerKBOYqV3lN9uk6G6nT\naNfuacAHgScA3H2dme1Zf5N8SXIKqMsfu5xFzy7qfbzDd/Q+VhAXkTKXAp8GeoDPAA8A17e4z28C\nX3P3B83spPDxeHe/CrgKwMxOBi6oDOAA7v63pftmdidwT4vtaa8shZLetu5PrJPkFrV2O4+hNctt\nz6FGQ/hf3N3NzAHMbI9Gf4CZdQDLgbXuPtnMzgNmEQwaGubum2ps901gEjAIeASYGbZhCcER6K3w\nqR919w2NtqcVSU0Bdcdzd9RcrhAuIiXuvgP43+GtbbsFhoT3hwLrqjznTOC2ejsxsyHABOCcNrat\nNVkKl7Xa2oTYx/v0F2CzEHJjbkva53tPe/uyptEQfruZ/SuwVzj6/lM0fpCfCazinYP4o8B9wJJa\nG5jZWIJSl6PCRb8Eji/b5hPuvrzBn595O3xHU8tFpFjMrIc6td/uflStdQ2YBTxkZlcTdIqMrfjZ\n7wJOBM7rZz+nAovd/fVqK81sBsHMLowYMaKF5qZAxOFy9+FbeWvDbjTTG77qiFE4fWf4iqxuvNEv\nOGkK3AlLy9z2taS9fVnUUAh396vD+sLXCerCv+Luj/S3nZl1EfRmXwFcGO7ryXBd3R9JUH+4K8Ex\nYxfgj420NY8G2aCqgXuQDaq5zUC/rcZ9SWYRaYvJ4f8/H/7/38L/n0UDAzPN7MfAflVW/TPB9IYX\nuPudZnYGcANwQtlzTgYerVaKUuFM6pTGuPt8YD5Ad3f3gAaT1iobrHxOLCIMlyMnvMLqn+wdBnGo\nFcbLj9uVAby0VSmIs9tujPr1yghaKyK1NDzdRxi6+w3eFa4BLuGdQTmN/qxlZvZTYD3BceI77l6e\nAm80s+3AncDl7t7ngJ2nXpXTDzt9p5rw8uXV1Pu2Wk3lh5ICt0i2uPsLAGb2d+7+wbJVXzKzJwhq\nxettf0KtdWZ2M8EZTYA76Bukp9J/Kcq+BFMnnlbvee1QlB65kRPKvvO0EPh7g/nWrTt1wOhzQCR6\ntbtSATP7k5m9XuvWz7aTgQ3uvqLZRpnZoQQj7buAA4EJZlYa3PMJdx8N/G14+8dq+3D3+e7e7e7d\nw4YNa7YJqfLlY7/MlMOn9PZ8D7JBTDl8StvqwfM2rZdIgZmZjSt7MJZ+jvMNWEdQDghBTffzZfsf\nGq7rb7Dlx4H73H1Li22RmOx0RrRWyFcpiUhL6vaEu/ueAGb2dYJe6X8j+OL8CYLBkfWMA04JR9N3\nAkPM7BZ3P6uBdp0GPObub4Q//0HgOOAX7r42bNufzOzfCXpXbm5gn5n25WO/rEGYItKfTwPfD8Ox\nAa8SjOFpxbnAteGc41sIzzCGTgMedvc3yzcwsweA6e5eGsQ5FZjbYjvaLwsDA0tabGu1qz43/fNF\npK2sSiVH3yeZ/drdP9DfsjrbjwcucvfJZctWA93VZkcxsykEB/4TCY4ZPyIobXkQ2MvdN5nZLgSn\nQH/s7t+r9/O7u7t9+fLCjOMcUM92UU7himSNma1w9+4mtxkK4O6ZTE5FO2bHYeSl93P/3Rf1hvBG\nwnjLJSlZ+YKTImmffSTt7UuDZo7ZjdaEv2lmnwAWEnyhPhN4s/4mNRt3PkGd+H7AU2b2gLtPN7Nu\n4LPuPh34AcFpz9KI/x+5+73h1IgPhQG8A/gx7Z2KS1JKg0VF+mdmX6l4DIC7X5ZIg6RhUYeb1XMn\nMTK8f+c9X2J33x79TOMK3E1Le6BNe/uyptGe8JHAtQQlJk4wzeAsd18dYdvapoi9Ks32hpf/YaXt\nm261AF6iIC5510yvipl9sexhJ8GsKavcvdWSlFi1dMzOYO9rveN1VMfeVR8YA1u31n2Ojq8izWt7\nT3gYtv++lUZJvGoduPsL2JoHVCS73P1b5Y/Dub0fSqg58UvRxXcuf+xy7njuDnb4DgbZIE4/7PRU\njespTUdYr5NDRKLVUAg3sxupMtds1npXRKeSRArmXQSzTEmMLn/s8p2mld3hO3ofpymIi0iyGp26\n6j7g/vC2mODql29E1SgREWmemfWY2VPh7TfAswSlhBKjO567o6nlSapVcqJSFJHoNVqOcmf5YzO7\njeBS8iIikh6Ty+5vA/7o7tuSakxRVbvCcb3lSWs6cDdYd68B9SL1NXzFzAp/BQxvZ0NEahn1zCod\nzEUac7m773QBMzP7t8pl0qQmB3sOskFVA3fpgmuVeqb1pG5APNQYQ/SHF6s/uaLuvlat+aojRunY\nLRJqtCb8T+xcE/4S8KVIWiSJivLDoJX96qCdf/qi1RZ/Xf4gvMDOMQm1JX5RXHxnAIM9Tz/s9J1q\nwsuX15J04K5Uc5D+wV30vLAm5tYkKIOz7Uh2NFqOsmfUDZHGRR1Wovgw0KwrUo96zVpjZrOBfwJ2\nN7PXS4uBvwDzE2tYElIQkEqDL9M8O8qAWeSzi6dHimbbSbs0nsnJgkZ7whe7+8T+lkn08hhW+pvT\nXH/IIvW5+5XAlWZ2pbvPTro9EgTxTIdu92IFbhkwdbINXN0QbmadBFNc7Wtm7+GdK90OAQ6MuG0i\ngP6QRfpjZke4+zPAHWZ2dOV6d38igWZJ0ZR6jtVLLNKQ/nrCPwPMAg4Ayg/irwPfiapRIiLSlAuB\nGcC3qqxzYEK8zZHMK11Nu7w33P2d5fXMGUoQG5Kh0gjJiroh3N2vBa41sy+4+7/E1CYREWmCu88I\n//+RpNuSO1EM9syAnhfWMPrgius8uQeDMkuvvVbNdIJUGiFZ0l85ygR3/wmw1sz+e+V6d78rspZJ\nrtSadUUENA1lu1Q7TgObgR533xB3e3Ij54G7llZmQcnF33RMX8DUc19c/ZWjHA/8BDi5yjoHFMJj\nluUDW7WDioK5lGTh33AGfBo4Dvhp+Hg8sAI4xMwuc/d/S6phkjFtCKBx/k2XPhdLE0M6MHX2QC+F\nUibiL2B56LlP6zz3WdBfOcpXw7uXufsfyteZ2SGRtUrqylNYaaSHXH/IIg0bDIxy9z8CmNl7gZuB\nDwM/BxTCpXEZOQNQ3jFVPp/Lwiu3tSeIS7/0OT0wjf7rvBOoHHH/A4p0EQiJjP54RdrmoFIAD20I\nl71iZm8n1SjJqRTXyxd9ckX1TGdDfzXhRxBcgW1oRa3hEKAzyoaJiEjTlpjZfcAd4eP/O1y2B/Ba\ncs2S3IohcLejBDPSAJqyLyJ5KHEpiv56wg8HJgN7sXNd+J+Ac6NqlIiIDMjnCYL3uPDxzcCd7u6A\nZk6RzKl3gbr+GDH0/uqqmtKC/mrC7wHuMbP/y91/Xr7OzMbV2ExERBIQhu0fhDcRSTkNaiy2RmvC\nr6FvTfi/VFkmklk6EErWhWWD3wCGE3QEGkE2H5Jow4ooZSUKRVXqMU/zhAb6nCmu/mrCjwPGAsPM\n7MKyVUOAjigbJrWlfYrCtLevGtXQSU58EzjZ3dP9B5d3KlGIXekzpl75Sto/h6R4+usJ3xV4d/i8\nPcuWvw58vJEfYGYdwHJgrbtPNrPzgFnA+4Bh7r6pxnbfBCYBg4BHgJnu7mZ2DLAA2B14oLS8kbbk\nQdoPMGltXyu93Oohlwz5owJ4A9RLLTmmEpfs6K8m/GfAz8xsgbu/MMCfMRNYRdB7DvAocB+wpNYG\nZjaWYGDRUeGiXxJcOGgJ8F2CQaGPE4TwE4EHB9g2KYAoernVQy4ptdzMFgF3A1tLC3V14zLqpU6v\nKu/NqGc2NzQIMzEpnaZRn0/Z0GhN+J/N7CqC6Qp7pyZ09wn1NjKzLoLe7CuAC8NtngzX1dvUw5+z\nK0FN4y7AH81sf2CIuz8W7uNm4FQUwkVEIOjs+DPw0bJlurpxo1IYpgqjzpejUc8E70GrYTyy+nD9\nG5EBajSE30pwNdjJwGeBacDGBra7BriEnUtZ+uXuy8zsp8B6ghD+HXdfZWbdwJqyp64BDqy2DzOb\nAcwAGDFJZTMMAAAd7ElEQVRiRDM/XqRpExdNZMOWDb2Ph3cOZ/GUxQm2SOrJ4riFRrj7OUm3IXfU\nQ54ao55Z1Za/3aTLI0VKBjX4vH3c/QbgbXf/mbt/CuivF3wysMHdVzTbKDM7FBgFdBGE7Alm9rfN\n7MPd57t7t7t3Dxs2rNkmiDSsMoADbNiygYmLJibUIqmnlXmH087Muszsh2a2IbzdGZ6RlDjVCu0K\n8y0b9cyq3tsZswdzxuzBjL5pdO9N4VqypNEQXrrc8Xozm2RmHwT27mebccApZrYaWEgQpG9p8Oed\nBjzm7m+4+xsE5SbHAWsJgnlJV7isMGodYNJy4El7+2pppX6uMoD3t1wkQjcC/wEcEN7uDZdJ3OZs\n7nuTtqk31qcU0kXSrtFylMvNbCjwRYL5wYcQzHBSk7vPBmYDmNl44CJ3P6vBn/cicK6ZXUlQjnI8\ncI27rzez183sWIKBmZ8M21MoaT+4pK19jY4UrxbEax3oRVJqmLuXh+4FZlb3WF04tQbSiYjErKEQ\n7u73hXc3E176eKAHdjM7n6BOfD/gKTN7wN2nh/Xen3X36QRXe5sA9BAMKvqRu98b7uJzvDNF4YNo\nUKY0YKA93ZrqSTLmZTM7C7gtfHwm8HKC7Umnar3SCubJSuksI03Lw2uQ2DTaE17NhQQDL/vl7ksI\npyR09+uA66o8ZzkwPby/HfhMjX0tB44cSINFBqK/wD28c3jV0pPhncOjapJILZ8iODs4j6ADYylw\ndpINyoy8hMAsa/Pvul0DORum6S+lSa2E8LpzDIoUxeIpizU7SobE/sEco/B6DqeULwvPWjbUYVJ4\nCkq5k4e/a8mvVkJ4Ya5SKdKfdgZulb9Er2AfzA2ftRTJCpUKSh7UDeFm9ieqh20jqMkWkTaK4uqe\nUng6aynRSbCEJ8ljYvnZNA9vUy89KFzg9Lywpup2IuX6u2x9UxfZEcmjNJYuqAdImqCzlhKNgtZA\nV34mlL7lLpy7namzg1g1+uAubv/GtnfWLmzj1To1diE3WilHEYlNUkG43oVd4g7ipbY4weVrHXoP\n+KDe8iLTWUuRZO10usmMhXO3h0t3PhHV8mdHQb/45FWjF+sRSUyer3DYqIVXbuu9b2W38uUQBPEx\nN42JtW2SPHff092HVLnt6e7qbBGJmZX9V6QWHZz7MfLS+/ssWz13UgItkSKrdiivdXjfzvadylXU\nOy4iIpI+6gmvo1oAr7dcpFW1AnMr/SmjbxrdexMRkfYr1YItunJbnzOUIrWoJ1wkZaoF8VVXtqf0\nRnXjItI2Bb3AUeW1Bpy+HSX1Ok5K2yY9wF+SpxCeQWmcrSOv0nxhl9K0WCIiicl54K6l9Bmw6ohR\nTQXwcgMapFnQLz55pRCeMWmarSMuSQfhNPxeq/0ODPiH2bsB29v2czT1oYhIe5QH9bZS4M4NhXDJ\nhDQE4aRV+x2sBMbcNIbtbQjiulCQiIhIfBTC61g9d5JmRymotJagVLNy2sre+xp8KSI1qYwhZfyd\n96TK+5ClzyEZGIXwfihwF0+WS37Ke6xVWiIivQp0kZdYjn3h73PUVFi1cD8qL8zT/2dFxaieivch\ny59D0jiFcJGcaueHjgJ9cZnZGOB7QCewDficu//KzC4GPhE+bTAwChjm7q9UbD8RuIpgStw3gLPd\n/XdxtV+KJZayuoovNKOmvlS2ru8Xmp3H9Hjv/3faTgpJITxjkh6kKFKiWvHC+CbwNXd/0MxOCh+P\nd/erCMI1ZnYycEFlAA99F/h7d19lZp8DvgycHU/TJVYqd6mp9zO61hmJJigD5IdCeAbpj02i0DOt\nRzXlUo0DQ8L7Q4F1VZ5zJnBbC9tL1hWo3CWNVKaSTQrhItKrWs/2QIJ5rfIVlbVk0izgITO7mqCk\nZGz5SjN7F3AicF6N7acDD5jZW8DrwLERtlWKqvQlYORBYK1cY1gkPgrhTaicCq6Djp1mpiiyVR8Y\nA1u3vrNgt90Y9ets/m5U8tOaejWZtZYriCfLzH4M7Fdl1T8DEwlKTe40szOAG4ATyp5zMvBojVIU\ngAuAk9z98bCO/NsEwbyyDTOAGQAjRowY8GuRGvJ8kZfy1+VhzXVag3iD70Otz6Ha6s+0IumkEN6P\ner2A29nOmJvGFD6I9wngAFu3suoDYzIdxEWKwt1PqLXOzG4GZoYP7wCur3jKVGqUopjZMOAD7v54\nuGgR8KMabZgPzAfo7u7WxWCjUIBw1vPCGkYf3PXOgjCMt/WLfqtfaBp8XrXPoYaCuUqAMiPyEG5m\nHcByYK27Tzaz8whOb76PYCT9pirbfASYV7boCGCqu99tZguA44HSv7Cz3T2SpNfIafh2XCQl8yoD\neH/LJVNURlJ46wiOuUuACcDzpRVmNjRcd1aNbV8FhprZYe7+HPB3gL7hSqR6XljzzoOowmiqQm7F\ndIeSGXH0hM8kOOiWBuY8CtxHcECvyt1/CowBMLO9gd8BD5c95WJ3/0EUjRWRvqIO3KWQr2CfSucC\n15rZYGALYclI6DTgYXd/s3wDM3sAmO7u68zsXOBOM9tBEMo/FVO7JU55LndJkYanO8x5aUpeSkYj\nDeFm1gVMAq4ALgRw9yfDdY3u5uPAg+7+5yjaKCLpofrw9HH3XwLH1Fi3AFhQZflJZfd/CPwwouZJ\nmuQ08KVNU9Md5rA0JU8XMhoU8f6vAS4BdrSwj2r1hleY2VNmNs/Mdqu2kZnNMLPlZrZ848aNLfz4\n+jroiGzfmbFb1beg9nLJtVohuplwPfqm0X1uIiJV1QqZOQufkj+R9YSb2WRgg7uvMLPxA9zH/sBo\n4KGyxbOBl4BdCQbxfAm4rHLbOAb5aHaUwKhfr8zV7CjSuv6CeL1QrVlURKRpRQzctUqAJDOiLEcZ\nB5wSXmGtExhiZre4e60BPNWcAfzQ3d8uLXD39eHdrWZ2I3BR21pcodEBaRq0hgK3iIhI3EpfPhTG\nMymyEO7uswl6rQl7wi9qMoBDcBW22eULzGx/d19vQVH5qcDTbWhuTf2F6bz02q0+5xzeWvZY7+Pd\njzuWkTfemGCLRERERPIr6prwPszsfDNbA3QBT5nZ9eHy7tL98PFI4CDgZxW7uNXMeoAeYF/g8jja\nnWeVARzgrWWPsfqccxJqkeRdO+rGRUQkVKC6+FqDL7M2KBNiuliPuy8hnJLQ3a8DrqvynOWUXUXN\n3VcDB1Z53oSImllYlQG8v+Ui7aDALSLSRhEF7jROB5j0z28XXTFTRFIlqosDaeyGiEhz8jQdYBop\nhGdQGr+VirRTu8NxXsZuiIhIfiiEZ0wU30p3P+7YqqUnux937ID2JzIQIy+9v8+y1XMnJdASERGR\n6CmECyNvvFGzo2RYHsJrtddQWp611yKSabr0vEhsFML70V8daVT1q3FT4M4mhVcRaZtac03n8NLn\nImmgEF5Ho3WkWQvcItKYPHzBFsmit99+mzVr1rBly5akm9Kwzs5Ourq62GWXXZJuStuMemaVxqFF\nSCFcRHJvIGesNJhTJDlr1qxhzz33ZOTIkQTX5ks3d+fll19mzZo1HHLIIUk3p60UuKOjEN6PhVdu\no/zP34Gps1v7tbXyrVLfSkUGRsFZJDu2bNmSmQAOYGbss88+bNy4MemmSIYohNdRCuCVh4CFV26D\naQPbZztmN1HglrxZPXdSZgeYTlw0kQ1bNvQ+Ht45nMVTFifYIpF8yEoAL8laeyV5CuF1VAvg+hOT\nNMlyeK2UxTZXBnCADVs2VF0O6o2XlJuzWbOjiMRIIVwk47IYXvOiWtCut1z15JJ6Cty9Ojo6GD36\nnbEhU6dO5dJLL02wRZI3CuEiIlXkZfpRkSK4+8m1XPXQs6x77S0O2Gt3Lv7Y4Zz6wQNb2ufuu+/O\nypUr29RCkb4UwuswgoGYlQMzVZIiUgwK3CLpd/eTa5l9Vw9vvb0dgLWvvcXsu4K/3VaDuEiUBiXd\ngDQb9cyqqjXhrQyMrLWtBluKiIg076qHnu0N4CVvvb2dqx56tqX9vvXWW4wZM6b3tmjRopb2J1JJ\nPeH9iCIcK3CLiIi0x7rX3mpqeaNUjiJRU0+4iMgA1SpXaXa5iAzcAXvt3tRykbRQT7iISAsUuEWS\ndfHHDt+pJhxg9106uPhjhyfYKpH+KYTnSNzzRevKnSIikrTS4Mt2z45SqgkvOfHEE5k7d25L+xQp\npxDehDRPV1YtgJeWRxHE23HlTxERkXY49YMHtn0mlO3bt/f/JJEWqCa8QdUCeL3lIiIiIiK1RN4T\nbmYdwHJgrbtPNrPzgFnA+4Bh7r6pyjYfAeaVLToCmOrud5vZIcBCYB9gBfCP7v6XqF+HiIiIiCQv\nL+WwcfSEzwTKfzOPAicAL9TawN1/6u5j3H0MMAH4M/BwuPobwDx3PxR4Ffh0JK0WERERkVSpVw6b\nNZGGcDPrAiYB15eWufuT7r66id18HHjQ3f9sZkYQyn8QrrsJOLVNzRURERERiUXUPeHXAJcAO1rY\nx1TgtvD+PsBr7r4tfLwGqDoSw8xmmNlyM1u+cePGFn58NtQafBnV7Ci68qeIiIjIwEVWE25mk4EN\n7r7CzMYPcB/7A6OBh5rd1t3nA/MBuru7fSA/v1zPtJ5YZ0cZyHSDUU5HWI0Ct4iIiMjARDkwcxxw\nipmdBHQCQ8zsFnc/q4l9nAH80N3fDh+/DOxlZoPD3vAuYG1bW11HXNMRxj3doIiIiOzMzLjwwgv5\n1re+BcDVV1/NG2+8wZw5c5JtmORGZOUo7j7b3bvcfSRBSclPmgzgAGfyTikK7u7ATwnqxAGmAfe0\nobmZsuqIUX1uIiIihfXU7TDvSJizV/D/p25veZe77bYbd911F5s29ZnETRKUp3LY2C/WY2bnE9SJ\n7wc8ZWYPuPt0M+sGPuvu08PnjQQOAn5WsYsvAQvN7HLgSeCGuNqeBnm5SE7cV/cUEZGceup2uPd8\nePut4PHm/wweAxx1xoB3O3jwYGbMmMG8efO44oor2tBQaZcs5Z16YrlYj7svcffJ4f3rwh7ywe5+\nQCl0u/vy0v3w8Wp3P9Ddd1Ts6/fu/iF3P9TdT3f3rXG8BmmfeuU2IiIiTVl82TsBvOTtt4LlLfr8\n5z/PrbfeyubNm1vel0glXTFTREREsmvzmuaWN2HIkCF88pOf5Lrrrmt5XyKVFMJTKO7pBttNNesi\nIhKboV3NLW/SrFmzuOGGG3jzzTfbsj+RkthrwqUxWQnclerVrP+3U6+OuTUiIpJ7E7+yc004wC67\nB8vbYO+99+aMM87ghhtu4FOf+lRb9ikCCuFtEef84aOeWVU16KZ9kELLE7WLiIhUUxp8ufiyoARl\naFcQwFsYlFnpi1/8It/5znfatj8RUAhvWbUAXloeZRDPo6z2/ouISMKOOqOtoRvgjTfe6L3/3ve+\nlz//+c9t3b+IQrgkTuFbRNoli2cKRaSYFMIHqFYPuFSnchQRiVperqMgIsWg2VEGQAG8tlofdJM0\nKFNERESkl3rCpe2qBfHV6CqZIiIiIiUK4RGJalBmlilwi4iIiAQUwttM4VtERERE+qOacBERqcnM\nxpjZY2a20syWm9mHwuUXh8tWmtnTZrbdzPausv0EM3sifM5NZhZZ50+tMSkalCkDdcUVV/DXf/3X\nHHXUUYwZM4bHH3886SZJjqgnfAB6pvXEeoEe0bRjIgn6JvA1d3/QzE4KH49396uAqwDM7GTgAnd/\npXxDMxsE3ARMdPfnzOwyYBpwQ1SN1XGhmO7//f1c+8S1vPTmS+y3x37MPHomk/5LayWQy5Yt4777\n7uOJJ55gt912Y9OmTfzlL39pU4tFFMIHTIE7Ppp2TCRRDgwJ7w8F1lV5zpnAbVWW7wP8xd2fCx8/\nAswmwhAuxXP/7+9nztI5bNm+BYD1b65nztI5AC0F8fXr17Pvvvuy2267AbDvvvu23FaRcgrhkirV\nZlB5MIF2iEivWcBDZnY1QQnj2PKVZvYu4ETgvCrbbgIGm1m3uy8HPg4cVO2HmNkMYAbAiBEj2td6\nyb1rn7i2N4CXbNm+hWufuLalEP7Rj36Uyy67jMMOO4wTTjiBKVOmcPzxx7faXJFeqgmX1KgWwEEX\n+hGJmpn9OKzZrrz9PfA/CEpNDgIuoG8v9snAo5WlKADu7sBUYJ6Z/Qr4E7C9Whvcfb67d7t797Bh\nw9r58iTnXnrzpaaWN+rd7343K1asYP78+QwbNowpU6awYMGClvYpUk494SIiBefuJ9RaZ2Y3AzPD\nh3cA11c8ZSrVS1FK+14G/G24r48Ch7XUWJEK++2xH+vfXF91eas6OjoYP34848ePZ/To0dx0002c\nffbZLe9XBNQTLhmgnnCRRK0DSufgJwDPl1aY2dBw3T21Njaz4eH/dwO+BHwvspZKIc08eiadHZ07\nLevs6GTm0TNrbNGYZ599luef7/3nzsqVKzn44INb2qdIOfWES+pNOvVqHrz7oj7LNShTJBbnAteG\nUwtuIazbDp0GPOzub5ZvYGYPANPdfR1wsZlNJuj0+a67/ySmdktBlOq+2z07yhtvvMEXvvAFXnvt\nNQYPHsyhhx7K/Pnz29FkESCGEG5mHcByYK27Tzaz8wgG+rwPGObum2psN4LgtOdBBJ2hJ7n7ajNb\nQNDzsjl86tnuvjLilyEJU+AWSYa7/xI4psa6BcCCKstPKrt/MXBxRM0TAYIg3mrornTMMcewdOnS\ntu5TpFwc5SgzgfIE9ShwAvBCP9vdDFzl7qOADwEbytZd7O5jwpsCeE7Uuqy9LncvIiIieRNpT7iZ\ndQGTgCuACwHc/clwXb3t3g8MdvdHwm3eiLKdkh4K3CIiIlIEUfeEXwNcAuxocrvDgNfM7C4ze9LM\nrgrLWkquMLOnzGxeONinDzObEV5iefnGjRsH2HwRERERkfaLrCc8HIizwd1XmNn4JjcfTDCl1QeB\nF4FFwNkE89POBl4CdgXmE4y2v6xyB+4+P1xPd3e3JtgQyZBqc8brLImIiORJlD3h44BTzGw1sBCY\nYGa3NLjtGmClu//e3bcBdwNHA7j7eg9sBW4kqBcXkZyoddGmWstFRESyKLIQ7u6z3b3L3UcSXMzh\nJ+5+VoOb/x9gLzMrXTZtAvBbADPbP/y/AacCT7e14SIiIiIiEYv9Yj1mdr6ZrQG6gKfM7PpweXfp\nvrtvBy4CFptZD2DA/w53cWu4rAfYF7g87tcgIiIi+XbBBRdwzTXX9D7+2Mc+xvTp03sff/GLX+Tb\n3/52Ek2TnIjlYj3uvgRYEt6/DriuynOWA9PLHj8CHFXleROiaqeIiGTbqiNG9Vmm6wzk3+Z772XD\nvGvYtn49g/ffn+EXzGLoySe3tM9x48Zx++23M2vWLHbs2MGmTZt4/fXXe9cvXbqUefPmtdp0KTBd\ntl5ERHKhWgCvt1zyYfO997L+f36FbevWgTvb1q1j/f/8Cpvvvbel/Y4dO5Zly5YB8Jvf/IYjjzyS\nPffck1dffZWtW7eyatUqjj766Ha8BCkohXARSRVdtElEmrFh3jX4li07LfMtW9gw75oaWzTmgAMO\nYPDgwbz44ossXbqU4447jg9/+MMsW7aM5cuXM3r0aHbdddeWfoYUWyzlKCIizVDgFpFGbVu/vqnl\nzRg7dixLly5l6dKlXHjhhaxdu5alS5cydOhQxo0b1/L+pdjUEy4iIiKZNXj//Zta3oxx48axdOlS\nenp6OPLIIzn22GNZtmwZS5cuZezYsS3vX4pNIVxEREQya/gFs7DOzp2WWWcnwy+Y1fK+x44dy333\n3cfee+9NR0cHe++9N6+99hrLli1TCJeWKYSLiEgu1JoFRbOj5NvQk09m/69fxuADDgAzBh9wAPt/\n/bKWZ0cBGD16NJs2beLYY4/dadnQoUPZd999W96/FJtqwkVEJDcUuItp6MkntyV0V+ro6NhpWkKA\nBQsWtP3nSDGpJ1xEREREJGYK4SIiIiIiMVMIFxERkdRx96Sb0JSstVeSpxAuIiIiqdLZ2cnLL7+c\nmWDr7rz88st0VszSIlKPBmaKiIhIqnR1dbFmzRo2btyYdFMa1tnZSVdXV9LNkAxRCBcREZFU2WWX\nXTjkkEOSboZIpFSOIiIiIiISM4VwEREREZGYKYSLiIiIiMTMsjLyuBVmthF4Iel2DNC+wKakG5EA\nve7iKOJrhsZf98HuPizqxqRJxo/ZUMx/00V8zaDXXSRtP2YXIoRnmZktd/fupNsRN73u4ijia4bi\nvu4iKOJ7W8TXDHrdSbcjTlG8ZpWjiIiIiIjETCFcRERERCRmCuHpNz/pBiREr7s4iviaobivuwiK\n+N4W8TWDXneRtP01qyZcRERERCRm6gkXEREREYmZQriIiIiISMwUwlPEzA4ys5+a2W/N7DdmNjNc\nvreZPWJmz4f/f0/SbW03M+swsyfN7L7w8SFm9riZ/c7MFpnZrkm3sd3MbC8z+4GZPWNmq8zsuIK8\n1xeE/76fNrPbzKwzj++3mX3fzDaY2dNly6q+vxa4Lnz9T5nZ0cm1XBpV5GM2FO+4rWO2jtntPmYr\nhKfLNuCL7v5+4Fjg82b2fuBSYLG7/xWwOHycNzOBVWWPvwHMc/dDgVeBTyfSqmhdC/zI3Y8APkDw\n+nP9XpvZgcD5QLe7Hwl0AFPJ5/u9ADixYlmt9/e/AX8V3mYA342pjdKaIh+zoXjHbR2zdcxu7zHb\n3XVL6Q24B/g74Flg/3DZ/sCzSbetza+zK/zHPQG4DzCCq1INDtcfBzyUdDvb/JqHAn8gHBxdtjzv\n7/WBwH8CewODw/f7Y3l9v4GRwNP9vb/AvwJnVnuebtm5FeWYHb6uQh23dczWMTuKY7Z6wlPKzEYC\nHwQeB97r7uvDVS8B702oWVG5BrgE2BE+3gd4zd23hY/XEBwI8uQQYCNwY3g693oz24Ocv9fuvha4\nGngRWA9sBlaQ//e7pNb7W/qgK8nz7yCXCnbMhuIdt3XM1jG77cdshfAUMrN3A3cCs9z99fJ1Hnzl\nys28kmY2Gdjg7iuSbkvMBgNHA9919w8Cb1JxGjNv7zVAWE/39wQfaAcAe9D39F8h5PH9LaoiHbOh\nsMdtHbN1zG77+6sQnjJmtgvBwfxWd78rXPxHM9s/XL8/sCGp9kVgHHCKma0GFhKc2rwW2MvMBofP\n6QLWJtO8yKwB1rj74+HjHxAc4PP8XgOcAPzB3Te6+9vAXQT/BvL+fpfUen/XAgeVPS/Pv4NcKeAx\nG4p53NYxW8fsth+zFcJTxMwMuAFY5e7fLlv1H8C08P40grrDXHD32e7e5e4jCQZ7/MTdPwH8FPh4\n+LRcvWYAd38J+E8zOzxcNBH4LTl+r0MvAsea2bvCf++l153r97tMrff3P4BPhiPujwU2l50ClZQq\n4jEbinnc1jFbx2wiOGbripkpYmZ/A/wC6OGdOrt/IqgxvB0YAbwAnOHuryTSyAiZ2XjgInefbGb/\nhaCHZW/gSeAsd9+aZPvazczGANcDuwK/B84h+GKc6/fazL4GTCGYWeJJYDpBLV2u3m8zuw0YD+wL\n/BH4KnA3Vd7f8MPtOwSnef8MnOPuy5NotzSu6MdsKNZxW8dsHbNp8zFbIVxEREREJGYqRxERERER\niZlCuIiIiIhIzBTCRURERERiphAuIiIiIhIzhXARERERkZgphItUMLPtZrbSzH5tZk+Y2dhw+Ugz\ne3qA+1xtZvu2t6UiIqJjtmTV4P6fIlI4b7n7GAAz+xhwJXB8sk0SEZEadMyWTFJPuEh9Q4BXKxea\nWaeZ3WhmPWb2pJl9JFzeYWZXm9nTZvaUmX2hYrvdzexBMzs3pvaLiBSJjtmSGeoJF+lrdzNbCXQC\n+wMTqjzn84C7+2gzOwJ42MwOI7iC2khgjLtvM7O9y7Z5N8EVxm5295sjfQUiIsWhY7ZkknrCRfp6\ny93HuPsRBJekvTm8RG25vwFuAXD3ZwguZ3sYcALwr+6+LVxXfvnie4AbdTAXEWkrHbMlkxTCRepw\n92XAvsCwNuzuUeDEKh8OIiLSBjpmS5YohIvUEZ627ABerlj1C+AT4XMOA0YAzwKPAJ8xs8HhuvJT\nm18hqFX8fyNutohIIemYLVmiEC7S1+7hdFcrgUXANHffXvGc/wUMMrOe8Dlnu/tW4HrgReApM/s1\n8A8V280M9//NaF+CiEhh6JgtmWTunnQbREREREQKRT3hIiIiIiIxUwgXEREREYmZQriIiIiISMwU\nwkVEREREYqYQLiIiIiISM4VwEREREZGYKYSLiIiIiMTs/wftaQnJLoYG/gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x112a40eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "groups = train[['Latitude','Longitude','Block','EWSN']].groupby('EWSN')\n",
    "\n",
    "fig = plt.figure(figsize=(12, 10))\n",
    "ax1 = fig.add_subplot(221)\n",
    "\n",
    "for label, df in groups:\n",
    "    ax1.scatter( df['Block'], df['Latitude'], label=label )\n",
    "plt.xlabel('Block')\n",
    "plt.ylabel('Lattitude')\n",
    "\n",
    "ax2 = fig.add_subplot(222)\n",
    "for label, df in groups:\n",
    "    ax2.scatter( df['Block'], df['Longitude'], label=label )\n",
    "plt.xlabel('Block')\n",
    "plt.ylabel('Longitude')\n",
    "plt.legend()\n",
    "#plt.scatter( train['Latitude'], train['Longitude'], ax=axes[0,1])\n",
    "#plt.scatter( train['Latitude'], train['Longitude'], ax=axes[1,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Conclusion: \n",
    "* if roads are north/south (green/orange), block is linearly related to lattitude\n",
    "* if roads as ease/west(blue, west), block is linearly related to longitude. \n",
    "So block is actually not very useful. All information are contained in longitude and lattitude. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Street</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "      <th>NumMosquitos</th>\n",
       "      <th>WnvPresent</th>\n",
       "      <th>Species2</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5023</th>\n",
       "      <td>2009-07-27</td>\n",
       "      <td>E 118TH ST</td>\n",
       "      <td>41.680946</td>\n",
       "      <td>-87.535198</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>CULEX PIPIENS</td>\n",
       "      <td>2009</td>\n",
       "      <td>7</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Date       Street   Latitude  Longitude  NumMosquitos  WnvPresent  \\\n",
       "5023  2009-07-27   E 118TH ST  41.680946 -87.535198             2           0   \n",
       "\n",
       "           Species2  Year  Month Week  \n",
       "5023  CULEX PIPIENS  2009      7   31  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop variables I am not going to use\n",
    "train = train.drop(['Address', 'AddressNumberAndStreet', 'Trap','AddressAccuracy', 'Species','EWSN', 'Block'], axis= 1)\n",
    "\n",
    "train.sample()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the number of traps\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">WnvPresent</th>\n",
       "      <th colspan=\"3\" halign=\"left\">NumMosquitos</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "      <th>count</th>\n",
       "      <th>sum</th>\n",
       "      <th>mean</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Street</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>N AVONDALE AVE</th>\n",
       "      <td>24</td>\n",
       "      <td>4</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>24</td>\n",
       "      <td>107</td>\n",
       "      <td>4.458333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W ROSCOE ST</th>\n",
       "      <td>14</td>\n",
       "      <td>2</td>\n",
       "      <td>0.142857</td>\n",
       "      <td>14</td>\n",
       "      <td>50</td>\n",
       "      <td>3.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N HARLEM AVE</th>\n",
       "      <td>50</td>\n",
       "      <td>7</td>\n",
       "      <td>0.140000</td>\n",
       "      <td>50</td>\n",
       "      <td>758</td>\n",
       "      <td>15.160000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W STRONG ST</th>\n",
       "      <td>30</td>\n",
       "      <td>4</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>30</td>\n",
       "      <td>364</td>\n",
       "      <td>12.133333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W 89TH ST</th>\n",
       "      <td>50</td>\n",
       "      <td>6</td>\n",
       "      <td>0.120000</td>\n",
       "      <td>50</td>\n",
       "      <td>578</td>\n",
       "      <td>11.560000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N MILWAUKEE AVE</th>\n",
       "      <td>122</td>\n",
       "      <td>14</td>\n",
       "      <td>0.114754</td>\n",
       "      <td>122</td>\n",
       "      <td>1346</td>\n",
       "      <td>11.032787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W FOSTER AVE</th>\n",
       "      <td>35</td>\n",
       "      <td>4</td>\n",
       "      <td>0.114286</td>\n",
       "      <td>35</td>\n",
       "      <td>169</td>\n",
       "      <td>4.828571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W 65TH ST</th>\n",
       "      <td>62</td>\n",
       "      <td>7</td>\n",
       "      <td>0.112903</td>\n",
       "      <td>62</td>\n",
       "      <td>557</td>\n",
       "      <td>8.983871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N OAK PARK AVE</th>\n",
       "      <td>216</td>\n",
       "      <td>24</td>\n",
       "      <td>0.111111</td>\n",
       "      <td>216</td>\n",
       "      <td>3940</td>\n",
       "      <td>18.240741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N LARAMIE AVE</th>\n",
       "      <td>65</td>\n",
       "      <td>7</td>\n",
       "      <td>0.107692</td>\n",
       "      <td>65</td>\n",
       "      <td>644</td>\n",
       "      <td>9.907692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S KOSTNER AVE</th>\n",
       "      <td>103</td>\n",
       "      <td>11</td>\n",
       "      <td>0.106796</td>\n",
       "      <td>103</td>\n",
       "      <td>2014</td>\n",
       "      <td>19.553398</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N OKETO AVE</th>\n",
       "      <td>66</td>\n",
       "      <td>7</td>\n",
       "      <td>0.106061</td>\n",
       "      <td>66</td>\n",
       "      <td>847</td>\n",
       "      <td>12.833333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N MONT CLARE AVE</th>\n",
       "      <td>87</td>\n",
       "      <td>9</td>\n",
       "      <td>0.103448</td>\n",
       "      <td>87</td>\n",
       "      <td>1281</td>\n",
       "      <td>14.724138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S NORDICA</th>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>20</td>\n",
       "      <td>90</td>\n",
       "      <td>4.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W HIGGINS AVE</th>\n",
       "      <td>31</td>\n",
       "      <td>3</td>\n",
       "      <td>0.096774</td>\n",
       "      <td>31</td>\n",
       "      <td>463</td>\n",
       "      <td>14.935484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N SPRINGFIELD AVE</th>\n",
       "      <td>62</td>\n",
       "      <td>6</td>\n",
       "      <td>0.096774</td>\n",
       "      <td>62</td>\n",
       "      <td>643</td>\n",
       "      <td>10.370968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W FULLERTON AVE</th>\n",
       "      <td>104</td>\n",
       "      <td>10</td>\n",
       "      <td>0.096154</td>\n",
       "      <td>104</td>\n",
       "      <td>1193</td>\n",
       "      <td>11.471154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N RIDGE AVE</th>\n",
       "      <td>42</td>\n",
       "      <td>4</td>\n",
       "      <td>0.095238</td>\n",
       "      <td>42</td>\n",
       "      <td>521</td>\n",
       "      <td>12.404762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N LAWLER AVE</th>\n",
       "      <td>21</td>\n",
       "      <td>2</td>\n",
       "      <td>0.095238</td>\n",
       "      <td>21</td>\n",
       "      <td>188</td>\n",
       "      <td>8.952381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>W 113TH ST</th>\n",
       "      <td>86</td>\n",
       "      <td>8</td>\n",
       "      <td>0.093023</td>\n",
       "      <td>86</td>\n",
       "      <td>1312</td>\n",
       "      <td>15.255814</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   WnvPresent               NumMosquitos                 \n",
       "                        count sum      mean        count   sum       mean\n",
       "Street                                                                   \n",
       " N AVONDALE AVE            24   4  0.166667           24   107   4.458333\n",
       " W ROSCOE ST               14   2  0.142857           14    50   3.571429\n",
       " N HARLEM AVE              50   7  0.140000           50   758  15.160000\n",
       " W STRONG ST               30   4  0.133333           30   364  12.133333\n",
       " W 89TH ST                 50   6  0.120000           50   578  11.560000\n",
       " N MILWAUKEE AVE          122  14  0.114754          122  1346  11.032787\n",
       " W FOSTER AVE              35   4  0.114286           35   169   4.828571\n",
       " W 65TH ST                 62   7  0.112903           62   557   8.983871\n",
       " N OAK PARK AVE           216  24  0.111111          216  3940  18.240741\n",
       " N LARAMIE AVE             65   7  0.107692           65   644   9.907692\n",
       " S KOSTNER AVE            103  11  0.106796          103  2014  19.553398\n",
       " N OKETO AVE               66   7  0.106061           66   847  12.833333\n",
       " N MONT CLARE AVE          87   9  0.103448           87  1281  14.724138\n",
       " S NORDICA                 20   2  0.100000           20    90   4.500000\n",
       " W HIGGINS AVE             31   3  0.096774           31   463  14.935484\n",
       " N SPRINGFIELD AVE         62   6  0.096774           62   643  10.370968\n",
       " W FULLERTON AVE          104  10  0.096154          104  1193  11.471154\n",
       " N RIDGE AVE               42   4  0.095238           42   521  12.404762\n",
       " N LAWLER AVE              21   2  0.095238           21   188   8.952381\n",
       " W 113TH ST                86   8  0.093023           86  1312  15.255814"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp = train[['WnvPresent','NumMosquitos', 'Street']].groupby('Street'). agg(\n",
    "    ['count', 'sum', 'mean' ]).sort_values([('WnvPresent', 'mean')], ascending=False)\n",
    "tmp.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tmp = train[['WnvPresent','NumMosquitos', 'Street']].groupby('Street'). agg(\n",
    "    ['count', 'sum', 'mean' ]).sort_values([('WnvPresent', 'mean')], ascending=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "41.644612 42.01743\n",
      "-87.930995 -87.531635\n"
     ]
    }
   ],
   "source": [
    "print( min( train['Latitude']), max(train['Latitude']) )\n",
    "print( min( train['Longitude']), max(train['Longitude']) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>NumMosquitos</th>\n",
       "      <th>WnvPresent</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th>Trap</th>\n",
       "      <th>Species</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">2007-05-29</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">T002</th>\n",
       "      <th>CULEX PIPIENS/RESTUANS</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CULEX RESTUANS</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>T007</th>\n",
       "      <th>CULEX RESTUANS</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">T015</th>\n",
       "      <th>CULEX PIPIENS/RESTUANS</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CULEX RESTUANS</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        NumMosquitos  WnvPresent\n",
       "Date       Trap Species                                         \n",
       "2007-05-29 T002 CULEX PIPIENS/RESTUANS             1           0\n",
       "                CULEX RESTUANS                     1           0\n",
       "           T007 CULEX RESTUANS                     1           0\n",
       "           T015 CULEX PIPIENS/RESTUANS             1           0\n",
       "                CULEX RESTUANS                     4           0"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_count = train[['Date','Trap', 'Species','NumMosquitos','WnvPresent']].groupby(\n",
    "    ['Date','Trap', 'Species']).sum()\n",
    "total_count.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Weather data:\n",
    "We only need the weather data \n",
    "We also need to calculate the distance of from traps to the two weather stations to determine weather data from which station should be used.\n",
    "\n",
    "*  weather.csv - weather data from 2007 to 2014. Column descriptions in noaa_weather_qclcd_documentation.pdf. \n",
    "   - Station 1: CHICAGO O'HARE INTERNATIONAL AIRPORT Lat: 41.995 Lon: -87.933 Elev: 662 ft. above sea level\n",
    "   - Station 2: CHICAGO MIDWAY INTL ARPT Lat: 41.786 Lon: -87.752 Elev: 612 ft. above sea level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2944, 22)\n",
      "Index(['Station', 'Date', 'Tmax', 'Tmin', 'Tavg', 'Depart', 'DewPoint',\n",
      "       'WetBulb', 'Heat', 'Cool', 'Sunrise', 'Sunset', 'CodeSum', 'Depth',\n",
      "       'Water1', 'SnowFall', 'PrecipTotal', 'StnPressure', 'SeaLevel',\n",
      "       'ResultSpeed', 'ResultDir', 'AvgSpeed'],\n",
      "      dtype='object')\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Station</th>\n",
       "      <th>Date</th>\n",
       "      <th>Tmax</th>\n",
       "      <th>Tmin</th>\n",
       "      <th>Tavg</th>\n",
       "      <th>Depart</th>\n",
       "      <th>DewPoint</th>\n",
       "      <th>WetBulb</th>\n",
       "      <th>Heat</th>\n",
       "      <th>Cool</th>\n",
       "      <th>...</th>\n",
       "      <th>CodeSum</th>\n",
       "      <th>Depth</th>\n",
       "      <th>Water1</th>\n",
       "      <th>SnowFall</th>\n",
       "      <th>PrecipTotal</th>\n",
       "      <th>StnPressure</th>\n",
       "      <th>SeaLevel</th>\n",
       "      <th>ResultSpeed</th>\n",
       "      <th>ResultDir</th>\n",
       "      <th>AvgSpeed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-01</td>\n",
       "      <td>83</td>\n",
       "      <td>50</td>\n",
       "      <td>67</td>\n",
       "      <td>14</td>\n",
       "      <td>51</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.10</td>\n",
       "      <td>29.82</td>\n",
       "      <td>1.7</td>\n",
       "      <td>27</td>\n",
       "      <td>9.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-01</td>\n",
       "      <td>84</td>\n",
       "      <td>52</td>\n",
       "      <td>68</td>\n",
       "      <td>M</td>\n",
       "      <td>51</td>\n",
       "      <td>57</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.18</td>\n",
       "      <td>29.82</td>\n",
       "      <td>2.7</td>\n",
       "      <td>25</td>\n",
       "      <td>9.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-02</td>\n",
       "      <td>59</td>\n",
       "      <td>42</td>\n",
       "      <td>51</td>\n",
       "      <td>-3</td>\n",
       "      <td>42</td>\n",
       "      <td>47</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>BR</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.38</td>\n",
       "      <td>30.09</td>\n",
       "      <td>13.0</td>\n",
       "      <td>4</td>\n",
       "      <td>13.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-02</td>\n",
       "      <td>60</td>\n",
       "      <td>43</td>\n",
       "      <td>52</td>\n",
       "      <td>M</td>\n",
       "      <td>42</td>\n",
       "      <td>47</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>BR HZ</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.44</td>\n",
       "      <td>30.08</td>\n",
       "      <td>13.3</td>\n",
       "      <td>2</td>\n",
       "      <td>13.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-03</td>\n",
       "      <td>66</td>\n",
       "      <td>46</td>\n",
       "      <td>56</td>\n",
       "      <td>2</td>\n",
       "      <td>40</td>\n",
       "      <td>48</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.39</td>\n",
       "      <td>30.12</td>\n",
       "      <td>11.7</td>\n",
       "      <td>7</td>\n",
       "      <td>11.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-03</td>\n",
       "      <td>67</td>\n",
       "      <td>48</td>\n",
       "      <td>58</td>\n",
       "      <td>M</td>\n",
       "      <td>40</td>\n",
       "      <td>50</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>HZ</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.46</td>\n",
       "      <td>30.12</td>\n",
       "      <td>12.9</td>\n",
       "      <td>6</td>\n",
       "      <td>13.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-04</td>\n",
       "      <td>66</td>\n",
       "      <td>49</td>\n",
       "      <td>58</td>\n",
       "      <td>4</td>\n",
       "      <td>41</td>\n",
       "      <td>50</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>RA</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>T</td>\n",
       "      <td>29.31</td>\n",
       "      <td>30.05</td>\n",
       "      <td>10.4</td>\n",
       "      <td>8</td>\n",
       "      <td>10.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-04</td>\n",
       "      <td>78</td>\n",
       "      <td>51</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>42</td>\n",
       "      <td>50</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.36</td>\n",
       "      <td>30.04</td>\n",
       "      <td>10.1</td>\n",
       "      <td>7</td>\n",
       "      <td>10.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-05</td>\n",
       "      <td>66</td>\n",
       "      <td>53</td>\n",
       "      <td>60</td>\n",
       "      <td>5</td>\n",
       "      <td>38</td>\n",
       "      <td>49</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>T</td>\n",
       "      <td>29.40</td>\n",
       "      <td>30.10</td>\n",
       "      <td>11.7</td>\n",
       "      <td>7</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-05</td>\n",
       "      <td>66</td>\n",
       "      <td>54</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>39</td>\n",
       "      <td>50</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>T</td>\n",
       "      <td>29.46</td>\n",
       "      <td>30.09</td>\n",
       "      <td>11.2</td>\n",
       "      <td>7</td>\n",
       "      <td>11.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-06</td>\n",
       "      <td>68</td>\n",
       "      <td>49</td>\n",
       "      <td>59</td>\n",
       "      <td>4</td>\n",
       "      <td>30</td>\n",
       "      <td>46</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.57</td>\n",
       "      <td>30.29</td>\n",
       "      <td>14.4</td>\n",
       "      <td>11</td>\n",
       "      <td>15.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-06</td>\n",
       "      <td>68</td>\n",
       "      <td>52</td>\n",
       "      <td>60</td>\n",
       "      <td>M</td>\n",
       "      <td>30</td>\n",
       "      <td>46</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.62</td>\n",
       "      <td>30.28</td>\n",
       "      <td>13.8</td>\n",
       "      <td>10</td>\n",
       "      <td>14.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-07</td>\n",
       "      <td>83</td>\n",
       "      <td>47</td>\n",
       "      <td>65</td>\n",
       "      <td>10</td>\n",
       "      <td>41</td>\n",
       "      <td>54</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>RA</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>T</td>\n",
       "      <td>29.38</td>\n",
       "      <td>30.12</td>\n",
       "      <td>8.6</td>\n",
       "      <td>18</td>\n",
       "      <td>10.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-07</td>\n",
       "      <td>84</td>\n",
       "      <td>50</td>\n",
       "      <td>67</td>\n",
       "      <td>M</td>\n",
       "      <td>39</td>\n",
       "      <td>53</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.44</td>\n",
       "      <td>30.12</td>\n",
       "      <td>8.5</td>\n",
       "      <td>17</td>\n",
       "      <td>9.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-08</td>\n",
       "      <td>82</td>\n",
       "      <td>54</td>\n",
       "      <td>68</td>\n",
       "      <td>12</td>\n",
       "      <td>58</td>\n",
       "      <td>62</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>BR</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.29</td>\n",
       "      <td>30.03</td>\n",
       "      <td>2.7</td>\n",
       "      <td>11</td>\n",
       "      <td>5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-08</td>\n",
       "      <td>80</td>\n",
       "      <td>60</td>\n",
       "      <td>70</td>\n",
       "      <td>M</td>\n",
       "      <td>57</td>\n",
       "      <td>63</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>HZ</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>T</td>\n",
       "      <td>29.36</td>\n",
       "      <td>30.02</td>\n",
       "      <td>2.5</td>\n",
       "      <td>8</td>\n",
       "      <td>5.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-09</td>\n",
       "      <td>77</td>\n",
       "      <td>61</td>\n",
       "      <td>69</td>\n",
       "      <td>13</td>\n",
       "      <td>59</td>\n",
       "      <td>63</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>BR HZ</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.13</td>\n",
       "      <td>29.21</td>\n",
       "      <td>29.94</td>\n",
       "      <td>3.9</td>\n",
       "      <td>9</td>\n",
       "      <td>6.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-09</td>\n",
       "      <td>76</td>\n",
       "      <td>63</td>\n",
       "      <td>70</td>\n",
       "      <td>M</td>\n",
       "      <td>60</td>\n",
       "      <td>63</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>BR HZ</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.02</td>\n",
       "      <td>29.28</td>\n",
       "      <td>29.93</td>\n",
       "      <td>3.9</td>\n",
       "      <td>7</td>\n",
       "      <td>5.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-10</td>\n",
       "      <td>84</td>\n",
       "      <td>56</td>\n",
       "      <td>70</td>\n",
       "      <td>14</td>\n",
       "      <td>52</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>BR</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.20</td>\n",
       "      <td>29.92</td>\n",
       "      <td>0.7</td>\n",
       "      <td>17</td>\n",
       "      <td>4.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-10</td>\n",
       "      <td>83</td>\n",
       "      <td>59</td>\n",
       "      <td>71</td>\n",
       "      <td>M</td>\n",
       "      <td>52</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>BR HZ</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.26</td>\n",
       "      <td>29.91</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9</td>\n",
       "      <td>3.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-11</td>\n",
       "      <td>70</td>\n",
       "      <td>51</td>\n",
       "      <td>61</td>\n",
       "      <td>4</td>\n",
       "      <td>42</td>\n",
       "      <td>51</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.33</td>\n",
       "      <td>30.04</td>\n",
       "      <td>11.3</td>\n",
       "      <td>3</td>\n",
       "      <td>12.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-11</td>\n",
       "      <td>73</td>\n",
       "      <td>49</td>\n",
       "      <td>61</td>\n",
       "      <td>M</td>\n",
       "      <td>44</td>\n",
       "      <td>51</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.39</td>\n",
       "      <td>30.03</td>\n",
       "      <td>11.7</td>\n",
       "      <td>36</td>\n",
       "      <td>12.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-12</td>\n",
       "      <td>64</td>\n",
       "      <td>46</td>\n",
       "      <td>55</td>\n",
       "      <td>-2</td>\n",
       "      <td>36</td>\n",
       "      <td>46</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.49</td>\n",
       "      <td>30.20</td>\n",
       "      <td>12.4</td>\n",
       "      <td>3</td>\n",
       "      <td>12.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-12</td>\n",
       "      <td>65</td>\n",
       "      <td>47</td>\n",
       "      <td>56</td>\n",
       "      <td>M</td>\n",
       "      <td>37</td>\n",
       "      <td>46</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.54</td>\n",
       "      <td>30.19</td>\n",
       "      <td>12.7</td>\n",
       "      <td>1</td>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-13</td>\n",
       "      <td>69</td>\n",
       "      <td>43</td>\n",
       "      <td>56</td>\n",
       "      <td>-2</td>\n",
       "      <td>33</td>\n",
       "      <td>46</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.49</td>\n",
       "      <td>30.24</td>\n",
       "      <td>6.6</td>\n",
       "      <td>14</td>\n",
       "      <td>8.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-13</td>\n",
       "      <td>69</td>\n",
       "      <td>44</td>\n",
       "      <td>57</td>\n",
       "      <td>M</td>\n",
       "      <td>32</td>\n",
       "      <td>46</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.55</td>\n",
       "      <td>30.24</td>\n",
       "      <td>6.4</td>\n",
       "      <td>11</td>\n",
       "      <td>7.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-14</td>\n",
       "      <td>90</td>\n",
       "      <td>56</td>\n",
       "      <td>73</td>\n",
       "      <td>15</td>\n",
       "      <td>47</td>\n",
       "      <td>59</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.23</td>\n",
       "      <td>29.97</td>\n",
       "      <td>16.9</td>\n",
       "      <td>21</td>\n",
       "      <td>17.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-14</td>\n",
       "      <td>90</td>\n",
       "      <td>54</td>\n",
       "      <td>72</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>58</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.31</td>\n",
       "      <td>29.98</td>\n",
       "      <td>14.1</td>\n",
       "      <td>21</td>\n",
       "      <td>14.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-15</td>\n",
       "      <td>80</td>\n",
       "      <td>57</td>\n",
       "      <td>69</td>\n",
       "      <td>11</td>\n",
       "      <td>56</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>RA BR</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.38</td>\n",
       "      <td>29.13</td>\n",
       "      <td>29.84</td>\n",
       "      <td>8.1</td>\n",
       "      <td>27</td>\n",
       "      <td>12.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-15</td>\n",
       "      <td>82</td>\n",
       "      <td>56</td>\n",
       "      <td>69</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>TSRA RA BR</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.60</td>\n",
       "      <td>29.19</td>\n",
       "      <td>29.83</td>\n",
       "      <td>8.1</td>\n",
       "      <td>25</td>\n",
       "      <td>10.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2914</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-17</td>\n",
       "      <td>62</td>\n",
       "      <td>48</td>\n",
       "      <td>55</td>\n",
       "      <td>3</td>\n",
       "      <td>46</td>\n",
       "      <td>50</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>T</td>\n",
       "      <td>28.99</td>\n",
       "      <td>29.69</td>\n",
       "      <td>11.1</td>\n",
       "      <td>29</td>\n",
       "      <td>13.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2915</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-17</td>\n",
       "      <td>63</td>\n",
       "      <td>49</td>\n",
       "      <td>56</td>\n",
       "      <td>M</td>\n",
       "      <td>48</td>\n",
       "      <td>52</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.01</td>\n",
       "      <td>29.05</td>\n",
       "      <td>29.69</td>\n",
       "      <td>10.7</td>\n",
       "      <td>28</td>\n",
       "      <td>12.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2916</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-18</td>\n",
       "      <td>50</td>\n",
       "      <td>36</td>\n",
       "      <td>43</td>\n",
       "      <td>-8</td>\n",
       "      <td>38</td>\n",
       "      <td>42</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>RA DZ</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>29.39</td>\n",
       "      <td>30.08</td>\n",
       "      <td>8.2</td>\n",
       "      <td>34</td>\n",
       "      <td>9.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2917</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-18</td>\n",
       "      <td>52</td>\n",
       "      <td>39</td>\n",
       "      <td>46</td>\n",
       "      <td>M</td>\n",
       "      <td>40</td>\n",
       "      <td>44</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>RA</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.02</td>\n",
       "      <td>29.44</td>\n",
       "      <td>30.08</td>\n",
       "      <td>7.6</td>\n",
       "      <td>34</td>\n",
       "      <td>8.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2918</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-19</td>\n",
       "      <td>54</td>\n",
       "      <td>33</td>\n",
       "      <td>44</td>\n",
       "      <td>-7</td>\n",
       "      <td>38</td>\n",
       "      <td>43</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.31</td>\n",
       "      <td>30.07</td>\n",
       "      <td>7.8</td>\n",
       "      <td>21</td>\n",
       "      <td>8.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2919</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-19</td>\n",
       "      <td>57</td>\n",
       "      <td>36</td>\n",
       "      <td>47</td>\n",
       "      <td>M</td>\n",
       "      <td>40</td>\n",
       "      <td>44</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.39</td>\n",
       "      <td>30.09</td>\n",
       "      <td>8.8</td>\n",
       "      <td>21</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2920</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-20</td>\n",
       "      <td>65</td>\n",
       "      <td>46</td>\n",
       "      <td>56</td>\n",
       "      <td>6</td>\n",
       "      <td>44</td>\n",
       "      <td>49</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.01</td>\n",
       "      <td>29.17</td>\n",
       "      <td>29.90</td>\n",
       "      <td>4.7</td>\n",
       "      <td>32</td>\n",
       "      <td>9.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2921</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-20</td>\n",
       "      <td>66</td>\n",
       "      <td>50</td>\n",
       "      <td>58</td>\n",
       "      <td>M</td>\n",
       "      <td>46</td>\n",
       "      <td>51</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.23</td>\n",
       "      <td>29.90</td>\n",
       "      <td>4.9</td>\n",
       "      <td>32</td>\n",
       "      <td>9.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2922</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-21</td>\n",
       "      <td>53</td>\n",
       "      <td>45</td>\n",
       "      <td>49</td>\n",
       "      <td>-1</td>\n",
       "      <td>39</td>\n",
       "      <td>44</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.46</td>\n",
       "      <td>30.16</td>\n",
       "      <td>10.8</td>\n",
       "      <td>1</td>\n",
       "      <td>11.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2923</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-21</td>\n",
       "      <td>54</td>\n",
       "      <td>49</td>\n",
       "      <td>52</td>\n",
       "      <td>M</td>\n",
       "      <td>41</td>\n",
       "      <td>46</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.51</td>\n",
       "      <td>30.15</td>\n",
       "      <td>11.0</td>\n",
       "      <td>36</td>\n",
       "      <td>11.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2924</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-22</td>\n",
       "      <td>56</td>\n",
       "      <td>35</td>\n",
       "      <td>46</td>\n",
       "      <td>-4</td>\n",
       "      <td>38</td>\n",
       "      <td>43</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.56</td>\n",
       "      <td>30.30</td>\n",
       "      <td>2.9</td>\n",
       "      <td>8</td>\n",
       "      <td>4.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2925</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-22</td>\n",
       "      <td>56</td>\n",
       "      <td>43</td>\n",
       "      <td>50</td>\n",
       "      <td>M</td>\n",
       "      <td>38</td>\n",
       "      <td>44</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.62</td>\n",
       "      <td>30.30</td>\n",
       "      <td>3.0</td>\n",
       "      <td>11</td>\n",
       "      <td>5.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2926</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-23</td>\n",
       "      <td>59</td>\n",
       "      <td>36</td>\n",
       "      <td>48</td>\n",
       "      <td>-1</td>\n",
       "      <td>40</td>\n",
       "      <td>45</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>BR</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.39</td>\n",
       "      <td>30.15</td>\n",
       "      <td>7.5</td>\n",
       "      <td>19</td>\n",
       "      <td>7.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2927</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-23</td>\n",
       "      <td>62</td>\n",
       "      <td>41</td>\n",
       "      <td>52</td>\n",
       "      <td>M</td>\n",
       "      <td>41</td>\n",
       "      <td>47</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>BR</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>T</td>\n",
       "      <td>29.46</td>\n",
       "      <td>30.16</td>\n",
       "      <td>7.9</td>\n",
       "      <td>19</td>\n",
       "      <td>8.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2928</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-24</td>\n",
       "      <td>67</td>\n",
       "      <td>52</td>\n",
       "      <td>60</td>\n",
       "      <td>11</td>\n",
       "      <td>46</td>\n",
       "      <td>52</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.25</td>\n",
       "      <td>30.00</td>\n",
       "      <td>4.4</td>\n",
       "      <td>22</td>\n",
       "      <td>5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2929</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-24</td>\n",
       "      <td>68</td>\n",
       "      <td>54</td>\n",
       "      <td>61</td>\n",
       "      <td>M</td>\n",
       "      <td>48</td>\n",
       "      <td>53</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.31</td>\n",
       "      <td>30.00</td>\n",
       "      <td>4.9</td>\n",
       "      <td>22</td>\n",
       "      <td>5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2930</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-25</td>\n",
       "      <td>72</td>\n",
       "      <td>49</td>\n",
       "      <td>61</td>\n",
       "      <td>13</td>\n",
       "      <td>40</td>\n",
       "      <td>51</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>BR</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.21</td>\n",
       "      <td>29.93</td>\n",
       "      <td>10.4</td>\n",
       "      <td>29</td>\n",
       "      <td>10.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2931</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-25</td>\n",
       "      <td>75</td>\n",
       "      <td>52</td>\n",
       "      <td>64</td>\n",
       "      <td>M</td>\n",
       "      <td>41</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>BR HZ</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.27</td>\n",
       "      <td>29.92</td>\n",
       "      <td>9.7</td>\n",
       "      <td>28</td>\n",
       "      <td>10.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2932</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-26</td>\n",
       "      <td>64</td>\n",
       "      <td>42</td>\n",
       "      <td>53</td>\n",
       "      <td>5</td>\n",
       "      <td>32</td>\n",
       "      <td>44</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.21</td>\n",
       "      <td>29.95</td>\n",
       "      <td>0.9</td>\n",
       "      <td>5</td>\n",
       "      <td>5.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2933</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-26</td>\n",
       "      <td>66</td>\n",
       "      <td>44</td>\n",
       "      <td>55</td>\n",
       "      <td>M</td>\n",
       "      <td>33</td>\n",
       "      <td>45</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.28</td>\n",
       "      <td>29.95</td>\n",
       "      <td>1.6</td>\n",
       "      <td>11</td>\n",
       "      <td>5.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2934</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-27</td>\n",
       "      <td>77</td>\n",
       "      <td>51</td>\n",
       "      <td>64</td>\n",
       "      <td>16</td>\n",
       "      <td>51</td>\n",
       "      <td>58</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>28.92</td>\n",
       "      <td>29.66</td>\n",
       "      <td>12.0</td>\n",
       "      <td>19</td>\n",
       "      <td>12.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2935</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-27</td>\n",
       "      <td>79</td>\n",
       "      <td>54</td>\n",
       "      <td>67</td>\n",
       "      <td>M</td>\n",
       "      <td>52</td>\n",
       "      <td>59</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>RA</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.02</td>\n",
       "      <td>29.00</td>\n",
       "      <td>29.67</td>\n",
       "      <td>12.7</td>\n",
       "      <td>19</td>\n",
       "      <td>13.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2936</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-28</td>\n",
       "      <td>68</td>\n",
       "      <td>45</td>\n",
       "      <td>57</td>\n",
       "      <td>10</td>\n",
       "      <td>38</td>\n",
       "      <td>47</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>T</td>\n",
       "      <td>29.15</td>\n",
       "      <td>29.85</td>\n",
       "      <td>14.8</td>\n",
       "      <td>26</td>\n",
       "      <td>15.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2937</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-28</td>\n",
       "      <td>66</td>\n",
       "      <td>48</td>\n",
       "      <td>57</td>\n",
       "      <td>M</td>\n",
       "      <td>40</td>\n",
       "      <td>48</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>RA</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.03</td>\n",
       "      <td>29.23</td>\n",
       "      <td>29.85</td>\n",
       "      <td>14.0</td>\n",
       "      <td>26</td>\n",
       "      <td>14.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2938</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-29</td>\n",
       "      <td>49</td>\n",
       "      <td>36</td>\n",
       "      <td>43</td>\n",
       "      <td>-4</td>\n",
       "      <td>32</td>\n",
       "      <td>40</td>\n",
       "      <td>22</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.36</td>\n",
       "      <td>30.06</td>\n",
       "      <td>9.5</td>\n",
       "      <td>29</td>\n",
       "      <td>9.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2939</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-29</td>\n",
       "      <td>49</td>\n",
       "      <td>40</td>\n",
       "      <td>45</td>\n",
       "      <td>M</td>\n",
       "      <td>34</td>\n",
       "      <td>42</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.42</td>\n",
       "      <td>30.07</td>\n",
       "      <td>8.5</td>\n",
       "      <td>29</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2940</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-30</td>\n",
       "      <td>51</td>\n",
       "      <td>32</td>\n",
       "      <td>42</td>\n",
       "      <td>-4</td>\n",
       "      <td>34</td>\n",
       "      <td>40</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td></td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>29.34</td>\n",
       "      <td>30.09</td>\n",
       "      <td>5.1</td>\n",
       "      <td>24</td>\n",
       "      <td>5.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2941</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-30</td>\n",
       "      <td>53</td>\n",
       "      <td>37</td>\n",
       "      <td>45</td>\n",
       "      <td>M</td>\n",
       "      <td>35</td>\n",
       "      <td>42</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>RA</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>T</td>\n",
       "      <td>29.41</td>\n",
       "      <td>30.10</td>\n",
       "      <td>5.9</td>\n",
       "      <td>23</td>\n",
       "      <td>6.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2942</th>\n",
       "      <td>1</td>\n",
       "      <td>2014-10-31</td>\n",
       "      <td>47</td>\n",
       "      <td>33</td>\n",
       "      <td>40</td>\n",
       "      <td>-6</td>\n",
       "      <td>25</td>\n",
       "      <td>33</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>RA SN</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.03</td>\n",
       "      <td>29.49</td>\n",
       "      <td>30.20</td>\n",
       "      <td>22.6</td>\n",
       "      <td>34</td>\n",
       "      <td>22.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2943</th>\n",
       "      <td>2</td>\n",
       "      <td>2014-10-31</td>\n",
       "      <td>49</td>\n",
       "      <td>34</td>\n",
       "      <td>42</td>\n",
       "      <td>M</td>\n",
       "      <td>29</td>\n",
       "      <td>36</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>RA SN BR</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>M</td>\n",
       "      <td>0.04</td>\n",
       "      <td>29.54</td>\n",
       "      <td>30.20</td>\n",
       "      <td>21.7</td>\n",
       "      <td>34</td>\n",
       "      <td>22.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2944 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Station       Date  Tmax  Tmin Tavg Depart  DewPoint WetBulb Heat Cool  \\\n",
       "0           1 2007-05-01    83    50   67     14        51      56    0    2   \n",
       "1           2 2007-05-01    84    52   68      M        51      57    0    3   \n",
       "2           1 2007-05-02    59    42   51     -3        42      47   14    0   \n",
       "3           2 2007-05-02    60    43   52      M        42      47   13    0   \n",
       "4           1 2007-05-03    66    46   56      2        40      48    9    0   \n",
       "5           2 2007-05-03    67    48   58      M        40      50    7    0   \n",
       "6           1 2007-05-04    66    49   58      4        41      50    7    0   \n",
       "7           2 2007-05-04    78    51    M      M        42      50    M    M   \n",
       "8           1 2007-05-05    66    53   60      5        38      49    5    0   \n",
       "9           2 2007-05-05    66    54   60      M        39      50    5    0   \n",
       "10          1 2007-05-06    68    49   59      4        30      46    6    0   \n",
       "11          2 2007-05-06    68    52   60      M        30      46    5    0   \n",
       "12          1 2007-05-07    83    47   65     10        41      54    0    0   \n",
       "13          2 2007-05-07    84    50   67      M        39      53    0    2   \n",
       "14          1 2007-05-08    82    54   68     12        58      62    0    3   \n",
       "15          2 2007-05-08    80    60   70      M        57      63    0    5   \n",
       "16          1 2007-05-09    77    61   69     13        59      63    0    4   \n",
       "17          2 2007-05-09    76    63   70      M        60      63    0    5   \n",
       "18          1 2007-05-10    84    56   70     14        52      60    0    5   \n",
       "19          2 2007-05-10    83    59   71      M        52      61    0    6   \n",
       "20          1 2007-05-11    70    51   61      4        42      51    4    0   \n",
       "21          2 2007-05-11    73    49   61      M        44      51    4    0   \n",
       "22          1 2007-05-12    64    46   55     -2        36      46   10    0   \n",
       "23          2 2007-05-12    65    47   56      M        37      46    9    0   \n",
       "24          1 2007-05-13    69    43   56     -2        33      46    9    0   \n",
       "25          2 2007-05-13    69    44   57      M        32      46    8    0   \n",
       "26          1 2007-05-14    90    56   73     15        47      59    0    8   \n",
       "27          2 2007-05-14    90    54   72      M        45      58    0    7   \n",
       "28          1 2007-05-15    80    57   69     11        56      61    0    4   \n",
       "29          2 2007-05-15    82    56   69      M        56      61    0    4   \n",
       "...       ...        ...   ...   ...  ...    ...       ...     ...  ...  ...   \n",
       "2914        1 2014-10-17    62    48   55      3        46      50   10    0   \n",
       "2915        2 2014-10-17    63    49   56      M        48      52    9    0   \n",
       "2916        1 2014-10-18    50    36   43     -8        38      42   22    0   \n",
       "2917        2 2014-10-18    52    39   46      M        40      44   19    0   \n",
       "2918        1 2014-10-19    54    33   44     -7        38      43   21    0   \n",
       "2919        2 2014-10-19    57    36   47      M        40      44   18    0   \n",
       "2920        1 2014-10-20    65    46   56      6        44      49    9    0   \n",
       "2921        2 2014-10-20    66    50   58      M        46      51    7    0   \n",
       "2922        1 2014-10-21    53    45   49     -1        39      44   16    0   \n",
       "2923        2 2014-10-21    54    49   52      M        41      46   13    0   \n",
       "2924        1 2014-10-22    56    35   46     -4        38      43   19    0   \n",
       "2925        2 2014-10-22    56    43   50      M        38      44   15    0   \n",
       "2926        1 2014-10-23    59    36   48     -1        40      45   17    0   \n",
       "2927        2 2014-10-23    62    41   52      M        41      47   13    0   \n",
       "2928        1 2014-10-24    67    52   60     11        46      52    5    0   \n",
       "2929        2 2014-10-24    68    54   61      M        48      53    4    0   \n",
       "2930        1 2014-10-25    72    49   61     13        40      51    4    0   \n",
       "2931        2 2014-10-25    75    52   64      M        41      52    1    0   \n",
       "2932        1 2014-10-26    64    42   53      5        32      44   12    0   \n",
       "2933        2 2014-10-26    66    44   55      M        33      45   10    0   \n",
       "2934        1 2014-10-27    77    51   64     16        51      58    1    0   \n",
       "2935        2 2014-10-27    79    54   67      M        52      59    0    2   \n",
       "2936        1 2014-10-28    68    45   57     10        38      47    8    0   \n",
       "2937        2 2014-10-28    66    48   57      M        40      48    8    0   \n",
       "2938        1 2014-10-29    49    36   43     -4        32      40   22    0   \n",
       "2939        2 2014-10-29    49    40   45      M        34      42   20    0   \n",
       "2940        1 2014-10-30    51    32   42     -4        34      40   23    0   \n",
       "2941        2 2014-10-30    53    37   45      M        35      42   20    0   \n",
       "2942        1 2014-10-31    47    33   40     -6        25      33   25    0   \n",
       "2943        2 2014-10-31    49    34   42      M        29      36   23    0   \n",
       "\n",
       "        ...        CodeSum Depth Water1 SnowFall PrecipTotal StnPressure  \\\n",
       "0       ...                    0      M      0.0        0.00       29.10   \n",
       "1       ...                    M      M        M        0.00       29.18   \n",
       "2       ...             BR     0      M      0.0        0.00       29.38   \n",
       "3       ...          BR HZ     M      M        M        0.00       29.44   \n",
       "4       ...                    0      M      0.0        0.00       29.39   \n",
       "5       ...             HZ     M      M        M        0.00       29.46   \n",
       "6       ...             RA     0      M      0.0           T       29.31   \n",
       "7       ...                    M      M        M        0.00       29.36   \n",
       "8       ...                    0      M      0.0           T       29.40   \n",
       "9       ...                    M      M        M           T       29.46   \n",
       "10      ...                    0      M      0.0        0.00       29.57   \n",
       "11      ...                    M      M        M        0.00       29.62   \n",
       "12      ...             RA     0      M      0.0           T       29.38   \n",
       "13      ...                    M      M        M        0.00       29.44   \n",
       "14      ...             BR     0      M      0.0        0.00       29.29   \n",
       "15      ...             HZ     M      M        M           T       29.36   \n",
       "16      ...          BR HZ     0      M      0.0        0.13       29.21   \n",
       "17      ...          BR HZ     M      M        M        0.02       29.28   \n",
       "18      ...             BR     0      M      0.0        0.00       29.20   \n",
       "19      ...          BR HZ     M      M        M        0.00       29.26   \n",
       "20      ...                    0      M      0.0        0.00       29.33   \n",
       "21      ...                    M      M        M        0.00       29.39   \n",
       "22      ...                    0      M      0.0        0.00       29.49   \n",
       "23      ...                    M      M        M        0.00       29.54   \n",
       "24      ...                    0      M      0.0        0.00       29.49   \n",
       "25      ...                    M      M        M        0.00       29.55   \n",
       "26      ...                    0      M      0.0        0.00       29.23   \n",
       "27      ...                    M      M        M        0.00       29.31   \n",
       "28      ...          RA BR     0      M      0.0        0.38       29.13   \n",
       "29      ...     TSRA RA BR     M      M        M        0.60       29.19   \n",
       "...     ...            ...   ...    ...      ...         ...         ...   \n",
       "2914    ...                    0      M      0.0           T       28.99   \n",
       "2915    ...                    M      M        M        0.01       29.05   \n",
       "2916    ...          RA DZ     0      M      0.0        0.01       29.39   \n",
       "2917    ...             RA     M      M        M        0.02       29.44   \n",
       "2918    ...                    0      M      0.0        0.00       29.31   \n",
       "2919    ...                    M      M        M        0.00       29.39   \n",
       "2920    ...                    0      M      0.0        0.01       29.17   \n",
       "2921    ...                    M      M        M        0.00       29.23   \n",
       "2922    ...                    0      M      0.0        0.00       29.46   \n",
       "2923    ...                    M      M        M        0.00       29.51   \n",
       "2924    ...                    0      M      0.0        0.00       29.56   \n",
       "2925    ...                    M      M        M        0.00       29.62   \n",
       "2926    ...             BR     0      M      0.0        0.00       29.39   \n",
       "2927    ...             BR     M      M        M           T       29.46   \n",
       "2928    ...                    0      M      0.0        0.00       29.25   \n",
       "2929    ...                    M      M        M        0.00       29.31   \n",
       "2930    ...             BR     0      M      0.0        0.00       29.21   \n",
       "2931    ...          BR HZ     M      M        M        0.00       29.27   \n",
       "2932    ...                    0      M      0.0        0.00       29.21   \n",
       "2933    ...                    M      M        M        0.00       29.28   \n",
       "2934    ...                    0      M      0.0        0.00       28.92   \n",
       "2935    ...             RA     M      M        M        0.02       29.00   \n",
       "2936    ...                    0      M      0.0           T       29.15   \n",
       "2937    ...             RA     M      M        M        0.03       29.23   \n",
       "2938    ...                    0      M      0.0        0.00       29.36   \n",
       "2939    ...                    M      M        M        0.00       29.42   \n",
       "2940    ...                    0      M      0.0        0.00       29.34   \n",
       "2941    ...             RA     M      M        M           T       29.41   \n",
       "2942    ...          RA SN     0      M      0.1        0.03       29.49   \n",
       "2943    ...       RA SN BR     M      M        M        0.04       29.54   \n",
       "\n",
       "     SeaLevel ResultSpeed ResultDir  AvgSpeed  \n",
       "0       29.82         1.7        27       9.2  \n",
       "1       29.82         2.7        25       9.6  \n",
       "2       30.09        13.0         4      13.4  \n",
       "3       30.08        13.3         2      13.4  \n",
       "4       30.12        11.7         7      11.9  \n",
       "5       30.12        12.9         6      13.2  \n",
       "6       30.05        10.4         8      10.8  \n",
       "7       30.04        10.1         7      10.4  \n",
       "8       30.10        11.7         7      12.0  \n",
       "9       30.09        11.2         7      11.5  \n",
       "10      30.29        14.4        11      15.0  \n",
       "11      30.28        13.8        10      14.5  \n",
       "12      30.12         8.6        18      10.5  \n",
       "13      30.12         8.5        17       9.9  \n",
       "14      30.03         2.7        11       5.8  \n",
       "15      30.02         2.5         8       5.4  \n",
       "16      29.94         3.9         9       6.2  \n",
       "17      29.93         3.9         7       5.9  \n",
       "18      29.92         0.7        17       4.1  \n",
       "19      29.91         2.0         9       3.9  \n",
       "20      30.04        11.3         3      12.9  \n",
       "21      30.03        11.7        36      12.8  \n",
       "22      30.20        12.4         3      12.9  \n",
       "23      30.19        12.7         1      13.0  \n",
       "24      30.24         6.6        14       8.1  \n",
       "25      30.24         6.4        11       7.6  \n",
       "26      29.97        16.9        21      17.3  \n",
       "27      29.98        14.1        21      14.6  \n",
       "28      29.84         8.1        27      12.3  \n",
       "29      29.83         8.1        25      10.8  \n",
       "...       ...         ...       ...       ...  \n",
       "2914    29.69        11.1        29      13.2  \n",
       "2915    29.69        10.7        28      12.7  \n",
       "2916    30.08         8.2        34       9.2  \n",
       "2917    30.08         7.6        34       8.7  \n",
       "2918    30.07         7.8        21       8.2  \n",
       "2919    30.09         8.8        21       9.0  \n",
       "2920    29.90         4.7        32       9.3  \n",
       "2921    29.90         4.9        32       9.1  \n",
       "2922    30.16        10.8         1      11.4  \n",
       "2923    30.15        11.0        36      11.6  \n",
       "2924    30.30         2.9         8       4.8  \n",
       "2925    30.30         3.0        11       5.1  \n",
       "2926    30.15         7.5        19       7.8  \n",
       "2927    30.16         7.9        19       8.4  \n",
       "2928    30.00         4.4        22       5.8  \n",
       "2929    30.00         4.9        22       5.8  \n",
       "2930    29.93        10.4        29      10.9  \n",
       "2931    29.92         9.7        28      10.4  \n",
       "2932    29.95         0.9         5       5.8  \n",
       "2933    29.95         1.6        11       5.4  \n",
       "2934    29.66        12.0        19      12.9  \n",
       "2935    29.67        12.7        19      13.6  \n",
       "2936    29.85        14.8        26      15.6  \n",
       "2937    29.85        14.0        26      14.6  \n",
       "2938    30.06         9.5        29       9.9  \n",
       "2939    30.07         8.5        29       9.0  \n",
       "2940    30.09         5.1        24       5.5  \n",
       "2941    30.10         5.9        23       6.5  \n",
       "2942    30.20        22.6        34      22.9  \n",
       "2943    30.20        21.7        34      22.6  \n",
       "\n",
       "[2944 rows x 22 columns]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(weather.shape)\n",
    "print(weather.columns)\n",
    "weather[\"Date\"] = weather[\"Date\"].astype(\"datetime64\")\n",
    "weather"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def process_line(line, indexes, weather_dic, weather_indexes):\n",
    "    def get(name):\n",
    "        return line[indexes[name]]\n",
    "\n",
    "    date = get(\"Date\")\n",
    "    month = float(date.split('-')[1])\n",
    "    week = int(date.split('-')[1]) * 4 + int(date.split('-')[2]) / 7\n",
    "    latitude = float(get(\"Latitude\"))\n",
    "    longitude = float(get(\"Longitude\"))\n",
    "    tmax = float(weather_dic[date][weather_indexes[\"Tmax\"]])\n",
    "    tmin = float(weather_dic[date][weather_indexes[\"Tmin\"]])\n",
    "    tavg = float(weather_dic[date][weather_indexes[\"Tavg\"]])\n",
    "    dewpoint = float(weather_dic[date][weather_indexes[\"DewPoint\"]])\n",
    "    wetbulb = float(weather_dic[date][weather_indexes[\"WetBulb\"]])\n",
    "    pressure = float(weather_dic[date][weather_indexes[\"StnPressure\"]])\n",
    "\n",
    "    return [month, week, latitude, longitude, tmax, tmin, tavg, dewpoint, wetbulb, pressure]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Missing data imputation for weather:\n",
    "* Code all missing data 'M' as np.nan\n",
    "* Replace all trace 'T' as 0.005"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Station</th>\n",
       "      <th>Date</th>\n",
       "      <th>Tmax</th>\n",
       "      <th>Tmin</th>\n",
       "      <th>Tavg</th>\n",
       "      <th>DewPoint</th>\n",
       "      <th>WetBulb</th>\n",
       "      <th>Heat</th>\n",
       "      <th>Cool</th>\n",
       "      <th>PrecipTotal</th>\n",
       "      <th>StnPressure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-01</td>\n",
       "      <td>83</td>\n",
       "      <td>50</td>\n",
       "      <td>67.0</td>\n",
       "      <td>51</td>\n",
       "      <td>56.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-01</td>\n",
       "      <td>84</td>\n",
       "      <td>52</td>\n",
       "      <td>68.0</td>\n",
       "      <td>51</td>\n",
       "      <td>57.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-02</td>\n",
       "      <td>59</td>\n",
       "      <td>42</td>\n",
       "      <td>51.0</td>\n",
       "      <td>42</td>\n",
       "      <td>47.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>2007-05-02</td>\n",
       "      <td>60</td>\n",
       "      <td>43</td>\n",
       "      <td>52.0</td>\n",
       "      <td>42</td>\n",
       "      <td>47.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2007-05-03</td>\n",
       "      <td>66</td>\n",
       "      <td>46</td>\n",
       "      <td>56.0</td>\n",
       "      <td>40</td>\n",
       "      <td>48.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29.39</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Station        Date  Tmax  Tmin  Tavg  DewPoint  WetBulb  Heat  Cool  \\\n",
       "0        1  2007-05-01    83    50  67.0        51     56.0   0.0   2.0   \n",
       "1        2  2007-05-01    84    52  68.0        51     57.0   0.0   3.0   \n",
       "2        1  2007-05-02    59    42  51.0        42     47.0  14.0   0.0   \n",
       "3        2  2007-05-02    60    43  52.0        42     47.0  13.0   0.0   \n",
       "4        1  2007-05-03    66    46  56.0        40     48.0   9.0   0.0   \n",
       "\n",
       "   PrecipTotal  StnPressure  \n",
       "0          0.0        29.10  \n",
       "1          0.0        29.18  \n",
       "2          0.0        29.38  \n",
       "3          0.0        29.44  \n",
       "4          0.0        29.39  "
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather = pd.read_csv(data_path + 'weather.csv')\n",
    "\n",
    "weather['PrecipTotal'] = weather['PrecipTotal'].replace('  T', 0.005)\n",
    "def convert_numeric(x):\n",
    "    weather[x] = pd.to_numeric(weather[x], errors= 'coerce')\n",
    "\n",
    "var_list = ['Tmax', 'Tmin', 'Tavg', 'DewPoint', 'WetBulb', 'Heat', 'Cool', 'PrecipTotal', 'StnPressure']\n",
    "for var in var_list:\n",
    "    convert_numeric(var)\n",
    "    \n",
    "weather = weather.drop(['CodeSum', 'Depart','Depth','SnowFall','Sunrise','Sunset','Water1', \n",
    "              'SeaLevel','ResultSpeed', 'ResultDir', 'AvgSpeed'], axis=1)\n",
    "\n",
    "weather.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take the mean of two station\n",
    "weather = weather.groupby('Date').mean().reset_index().drop('Station', axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Tmax</th>\n",
       "      <th>Tmin</th>\n",
       "      <th>Tavg</th>\n",
       "      <th>DewPoint</th>\n",
       "      <th>WetBulb</th>\n",
       "      <th>Heat</th>\n",
       "      <th>Cool</th>\n",
       "      <th>PrecipTotal</th>\n",
       "      <th>StnPressure</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2007-05-01</td>\n",
       "      <td>83.5</td>\n",
       "      <td>51.0</td>\n",
       "      <td>67.5</td>\n",
       "      <td>51.0</td>\n",
       "      <td>56.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>29.140</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2007-05-02</td>\n",
       "      <td>59.5</td>\n",
       "      <td>42.5</td>\n",
       "      <td>51.5</td>\n",
       "      <td>42.0</td>\n",
       "      <td>47.0</td>\n",
       "      <td>13.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>29.410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2007-05-03</td>\n",
       "      <td>66.5</td>\n",
       "      <td>47.0</td>\n",
       "      <td>57.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>49.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>29.425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2007-05-04</td>\n",
       "      <td>72.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>41.5</td>\n",
       "      <td>50.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0025</td>\n",
       "      <td>29.335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2007-05-05</td>\n",
       "      <td>66.0</td>\n",
       "      <td>53.5</td>\n",
       "      <td>60.0</td>\n",
       "      <td>38.5</td>\n",
       "      <td>49.5</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0050</td>\n",
       "      <td>29.430</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date  Tmax  Tmin  Tavg  DewPoint  WetBulb  Heat  Cool  PrecipTotal  \\\n",
       "0  2007-05-01  83.5  51.0  67.5      51.0     56.5   0.0   2.5       0.0000   \n",
       "1  2007-05-02  59.5  42.5  51.5      42.0     47.0  13.5   0.0       0.0000   \n",
       "2  2007-05-03  66.5  47.0  57.0      40.0     49.0   8.0   0.0       0.0000   \n",
       "3  2007-05-04  72.0  50.0  58.0      41.5     50.0   7.0   0.0       0.0025   \n",
       "4  2007-05-05  66.0  53.5  60.0      38.5     49.5   5.0   0.0       0.0050   \n",
       "\n",
       "   StnPressure  \n",
       "0       29.140  \n",
       "1       29.410  \n",
       "2       29.425  \n",
       "3       29.335  \n",
       "4       29.430  "
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Spray Data\n",
    "The City of Chicago also does spraying to kill mosquitos. You are given the GIS data for their spray efforts in 2011 and 2013. Spraying can reduce the number of mosquitos in the area, and therefore might eliminate the appearance of West Nile virus. \n",
    "\n",
    "*  spray.csv - GIS data of spraying efforts in 2011 and 2013\n",
    "   - Date, Time: the date and time of the spray\n",
    "   - Latitude, Longitude: the Latitude and Longitude of the spray\n",
    "   \n",
    "* From analysis below, we can see that there are some spray data in Aug and Sep of 2011, and more data in July, Aug, Sep of 2013. \n",
    "* The spray data can be match to the main data using Date and approximate Lattitude and Longitude. \n",
    "* Time will not be used as the trap data only have date."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(14835, 4)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Time</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-08-29</td>\n",
       "      <td>6:56:58 PM</td>\n",
       "      <td>42.391623</td>\n",
       "      <td>-88.089163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-08-29</td>\n",
       "      <td>6:57:08 PM</td>\n",
       "      <td>42.391348</td>\n",
       "      <td>-88.089163</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         Date        Time   Latitude  Longitude\n",
       "0  2011-08-29  6:56:58 PM  42.391623 -88.089163\n",
       "1  2011-08-29  6:57:08 PM  42.391348 -88.089163"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(spray.shape)\n",
    "spray[\"Date\"] = spray[\"Date\"].astype(\"datetime64\")\n",
    "spray.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date           0\n",
       "Time         584\n",
       "Latitude       0\n",
       "Longitude      0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check counts for missing values in each column\n",
    "spray.isnull().sum() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2011-08-29 00:00:00 2013-09-05 00:00:00\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x11794ae80>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEuCAYAAACH0cUhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGiBJREFUeJzt3Xu0nXV95/H3RyJi0eEiR8QkNlTjtaOIKeiyzqiM4eYY\nxiUUp9XUZibWQad2OZ3Grq7FeKGDM10ysipMWUIN6oCMl0WWUGmKOJ1OByRcRC4ikYsk5XJKAl5Q\nR/A7f+xf2DvpOZ59yMneOed5v9Y66zzP7/k9+/k935ycz3kuez+pKiRJ3fOUcQ9AkjQeBoAkdZQB\nIEkdZQBIUkcZAJLUUQaAJHWUASBJHWUASFJHGQCS1FEGgCR11KJxD+AXOeSQQ2rZsmXjHoYkzSvX\nXXfdP1TVxEz99uoAWLZsGZs2bRr3MCRpXklyzzD9PAUkSR01VAAk+f0ktyS5OclFSfZLcniSa5Js\nTvL5JPu2vk9r85vb8mUDr/PB1n57kmP3zC5JkoYxYwAkWQz8e2BFVf0qsA9wKvAx4KyqegGwHVjT\nVlkDbG/tZ7V+JHlpW+9lwHHAOUn2mdvdkSQNa9hTQIuApydZBPwScB/wRuALbfl64KQ2varN05Yf\nkySt/eKq+mlV3QVsBo7a/V2QJD0ZMwZAVW0F/hT4Hr1f/I8A1wEPV9VjrdsWYHGbXgzc29Z9rPV/\n1mD7FOtIkkZsmFNAB9H76/1w4LnA/vRO4ewRSdYm2ZRk0+Tk5J7ajCR13jCngP4FcFdVTVbVz4Av\nAa8FDmynhACWAFvb9FZgKUBbfgDw0GD7FOs8oarOq6oVVbViYmLG21glSU/SMAHwPeDVSX6pncs/\nBrgVuAp4W+uzGri0TW9o87TlX6veg4c3AKe2u4QOB5YD35ib3ZAkzdaMbwSrqmuSfAG4HngMuAE4\nD7gMuDjJR1vb+W2V84HPJNkMbKN35w9VdUuSS+iFx2PAaVX1+Bzvj6RZWrbusnEPgbvPPHHcQ+ik\nod4JXFWnA6fv0nwnU9zFU1U/AU6e5nXOAM6Y5RglSXuA7wSWpI4yACSpowwASeooA0CSOsoAkKSO\nMgAkqaMMAEnqKANAkjrKAJCkjjIAJKmjDABJ6igDQJI6ygCQpI4yACSpowwASeooA0CSOsoAkKSO\nmjEAkrwoyY0DX99P8v4kByfZmOSO9v2g1j9Jzk6yOclNSY4ceK3Vrf8dSVZPv1VJ0p42YwBU1e1V\ndURVHQG8CngU+DKwDriyqpYDV7Z5gOPpPfB9ObAWOBcgycH0Hit5NL1HSZ6+IzQkSaM321NAxwDf\nrap7gFXA+ta+HjipTa8CLqyeq4EDkxwGHAtsrKptVbUd2Agct9t7IEl6UmYbAKcCF7XpQ6vqvjZ9\nP3Bom14M3DuwzpbWNl27JGkMhg6AJPsCbwH+567LqqqAmosBJVmbZFOSTZOTk3PxkpKkKczmCOB4\n4PqqeqDNP9BO7dC+P9jatwJLB9Zb0tqma99JVZ1XVSuqasXExMQshidJmo3ZBMDb6Z/+AdgA7LiT\nZzVw6UD7O9vdQK8GHmmniq4AViY5qF38XdnaJEljsGiYTkn2B94EvHug+UzgkiRrgHuAU1r75cAJ\nwGZ6dwy9C6CqtiX5CHBt6/fhqtq223sgSXpShgqAqvoR8Kxd2h6id1fQrn0LOG2a17kAuGD2w5Qk\nzTXfCSxJHWUASFJHGQCS1FEGgCR1lAEgSR1lAEhSRxkAktRRBoAkdZQBIEkdZQBIUkcZAJLUUQaA\nJHWUASBJHWUASFJHGQCS1FEGgCR1lAEgSR1lAEhSRw0VAEkOTPKFJN9OcluS1yQ5OMnGJHe07we1\nvklydpLNSW5KcuTA66xu/e9Isnr6LUqS9rRhjwA+AXy1ql4MvAK4DVgHXFlVy4Er2zzA8cDy9rUW\nOBcgycHA6cDRwFHA6TtCQ5I0ejMGQJIDgH8GnA9QVf+vqh4GVgHrW7f1wEltehVwYfVcDRyY5DDg\nWGBjVW2rqu3ARuC4Od0bSdLQhjkCOByYBP4iyQ1JPpVkf+DQqrqv9bkfOLRNLwbuHVh/S2ubrl2S\nNAbDBMAi4Ejg3Kp6JfAj+qd7AKiqAmouBpRkbZJNSTZNTk7OxUtKkqYwTABsAbZU1TVt/gv0AuGB\ndmqH9v3BtnwrsHRg/SWtbbr2nVTVeVW1oqpWTExMzGZfJEmzMGMAVNX9wL1JXtSajgFuBTYAO+7k\nWQ1c2qY3AO9sdwO9GniknSq6AliZ5KB28Xdla5MkjcGiIfu9D/hckn2BO4F30QuPS5KsAe4BTml9\nLwdOADYDj7a+VNW2JB8Brm39PlxV2+ZkLyRJszZUAFTVjcCKKRYdM0XfAk6b5nUuAC6YzQAlSXuG\n7wSWpI4yACSpowwASeooA0CSOsoAkKSOMgAkqaMMAEnqKANAkjrKAJCkjjIAJKmjDABJ6igDQJI6\nygCQpI4yACSpowwASeooA0CSOsoAkKSOGioAktyd5FtJbkyyqbUdnGRjkjva94Nae5KcnWRzkpuS\nHDnwOqtb/zuSrJ5ue5KkPW82RwBvqKojqmrHoyHXAVdW1XLgyjYPcDywvH2tBc6FXmAApwNHA0cB\np+8IDUnS6O3OKaBVwPo2vR44aaD9wuq5GjgwyWHAscDGqtpWVduBjcBxu7F9SdJuGDYACvirJNcl\nWdvaDq2q+9r0/cChbXoxcO/Aulta23TtkqQxWDRkv1+vqq1Jng1sTPLtwYVVVUlqLgbUAmYtwPOe\n97y5eElJ0hSGOgKoqq3t+4PAl+mdw3+gndqhfX+wdd8KLB1YfUlrm659122dV1UrqmrFxMTE7PZG\nkjS0GQMgyf5JnrljGlgJ3AxsAHbcybMauLRNbwDe2e4GejXwSDtVdAWwMslB7eLvytYmSRqDYU4B\nHQp8OcmO/v+jqr6a5FrgkiRrgHuAU1r/y4ETgM3Ao8C7AKpqW5KPANe2fh+uqm1ztieSpFmZMQCq\n6k7gFVO0PwQcM0V7AadN81oXABfMfpiSpLnmO4ElqaMMAEnqKANAkjrKAJCkjjIAJKmjDABJ6igD\nQJI6ygCQpI4yACSpowwASeooA0CSOsoAkKSOMgAkqaMMAEnqKANAkjrKAJCkjjIAJKmjhg6AJPsk\nuSHJV9r84UmuSbI5yeeT7Nvan9bmN7flywZe44Ot/fYkx871zkiShjebI4DfA24bmP8YcFZVvQDY\nDqxp7WuA7a39rNaPJC8FTgVeBhwHnJNkn90bviTpyRoqAJIsAU4EPtXmA7wR+ELrsh44qU2vavO0\n5ce0/quAi6vqp1V1F72Hxh81FzshSZq9YY8A/hvwH4Gft/lnAQ9X1WNtfguwuE0vBu4FaMsfaf2f\naJ9iHUnSiM0YAEneDDxYVdeNYDwkWZtkU5JNk5OTo9ikJHXSMEcArwXekuRu4GJ6p34+ARyYZFHr\nswTY2qa3AksB2vIDgIcG26dY5wlVdV5VraiqFRMTE7PeIUnScGYMgKr6YFUtqapl9C7ifq2qfhO4\nCnhb67YauLRNb2jztOVfq6pq7ae2u4QOB5YD35izPZEkzcqimbtM6w+Bi5N8FLgBOL+1nw98Jslm\nYBu90KCqbklyCXAr8BhwWlU9vhvblyTthlkFQFV9Hfh6m76TKe7iqaqfACdPs/4ZwBmzHaQkae7t\nzhGANG8tW3fZuIfA3WeeOO4hqOP8KAhJ6igDQJI6ygCQpI4yACSpowwASeooA0CSOsoAkKSOMgAk\nqaMMAEnqKANAkjrKAJCkjjIAJKmjDABJ6igDQJI6ygCQpI4yACSpo2YMgCT7JflGkm8muSXJh1r7\n4UmuSbI5yeeT7Nvan9bmN7flywZe64Ot/fYkx+6pnZIkzWyYI4CfAm+sqlcARwDHJXk18DHgrKp6\nAbAdWNP6rwG2t/azWj+SvJTe84FfBhwHnJNkn7ncGUnS8GYMgOr5YZt9avsq4I3AF1r7euCkNr2q\nzdOWH5Mkrf3iqvppVd0FbGaKZwpLkkZjqGsASfZJciPwILAR+C7wcFU91rpsARa36cXAvQBt+SPA\nswbbp1hHkjRiQwVAVT1eVUcAS+j91f7iPTWgJGuTbEqyaXJyck9tRpI6b1Z3AVXVw8BVwGuAA5Ms\naouWAFvb9FZgKUBbfgDw0GD7FOsMbuO8qlpRVSsmJiZmMzxJ0iwMcxfQRJID2/TTgTcBt9ELgre1\nbquBS9v0hjZPW/61qqrWfmq7S+hwYDnwjbnaEUnS7CyauQuHAevbHTtPAS6pqq8kuRW4OMlHgRuA\n81v/84HPJNkMbKN35w9VdUuSS4BbgceA06rq8bndHUnSsGYMgKq6CXjlFO13MsVdPFX1E+DkaV7r\nDOCM2Q9TkjTXfCewJHWUASBJHWUASFJHGQCS1FEGgCR1lAEgSR1lAEhSRxkAktRRBoAkdZQBIEkd\nZQBIUkcZAJLUUQaAJHWUASBJHWUASFJHGQCS1FEGgCR1lAEgSR01zEPhlya5KsmtSW5J8nut/eAk\nG5Pc0b4f1NqT5Owkm5PclOTIgdda3frfkWT1dNuUJO15wzwU/jHgA1V1fZJnAtcl2Qj8NnBlVZ2Z\nZB2wDvhD4Hhgefs6GjgXODrJwcDpwAqg2utsqKrtc71TmtqydZeNewgA3H3mieMegiSGOAKoqvuq\n6vo2/QPgNmAxsApY37qtB05q06uAC6vnauDAJIcBxwIbq2pb+6W/EThuTvdGkjS0WV0DSLIMeCVw\nDXBoVd3XFt0PHNqmFwP3Dqy2pbVN1y5JGoOhAyDJM4AvAu+vqu8PLquqondaZ7clWZtkU5JNk5OT\nc/GSkqQpDBUASZ5K75f/56rqS635gXZqh/b9wda+FVg6sPqS1jZd+06q6ryqWlFVKyYmJmazL5Kk\nWRjmLqAA5wO3VdXHBxZtAHbcybMauHSg/Z3tbqBXA4+0U0VXACuTHNTuGFrZ2iRJYzDMXUCvBd4B\nfCvJja3tj4AzgUuSrAHuAU5pyy4HTgA2A48C7wKoqm1JPgJc2/p9uKq2zcleSJJmbcYAqKq/BTLN\n4mOm6F/AadO81gXABbMZoCRpz/CdwJLUUQaAJHWUASBJHWUASFJHGQCS1FEGgCR1lAEgSR1lAEhS\nRxkAktRRBoAkdZQBIEkdZQBIUkcZAJLUUQaAJHWUASBJHWUASFJHGQCS1FHDPBP4giQPJrl5oO3g\nJBuT3NG+H9Tak+TsJJuT3JTkyIF1Vrf+dyRZPdW2JEmjM8wRwKeB43ZpWwdcWVXLgSvbPMDxwPL2\ntRY4F3qBAZwOHA0cBZy+IzQkSeMxYwBU1d8Auz68fRWwvk2vB04aaL+weq4GDkxyGHAssLGqtlXV\ndmAj/zhUJEkj9GSvARxaVfe16fuBQ9v0YuDegX5bWtt07ZKkMdnti8BVVUDNwVgASLI2yaYkmyYn\nJ+fqZSVJu3iyAfBAO7VD+/5ga98KLB3ot6S1Tdf+j1TVeVW1oqpWTExMPMnhSZJmsuhJrrcBWA2c\n2b5fOtD+3iQX07vg+0hV3ZfkCuBPBi78rgQ++OSHLUlzb9m6y8Y9BO4+88SRbWvGAEhyEfB64JAk\nW+jdzXMmcEmSNcA9wCmt++XACcBm4FHgXQBVtS3JR4BrW78PV9WuF5YlSSM0YwBU1dunWXTMFH0L\nOG2a17kAuGBWo5Mk7TG+E1iSOsoAkKSOMgAkqaMMAEnqKANAkjrKAJCkjjIAJKmjDABJ6igDQJI6\nygCQpI4yACSpowwASeooA0CSOsoAkKSOMgAkqaMMAEnqKANAkjpq5AGQ5LgktyfZnGTdqLcvSeoZ\naQAk2Qf4JHA88FLg7UleOsoxSJJ6Znwm8Bw7CthcVXcCJLkYWAXcuqc2uGzdZXvqpWfl7jNPHPcQ\nJGknoz4FtBi4d2B+S2uTJI1Yqmp0G0veBhxXVf+mzb8DOLqq3jvQZy2wts2+CLh9ZAOc3iHAP4x7\nEHsJa9FnLfqsRd/eUItfrqqJmTqN+hTQVmDpwPyS1vaEqjoPOG+Ug5pJkk1VtWLc49gbWIs+a9Fn\nLfrmUy1GfQroWmB5ksOT7AucCmwY8RgkSYz4CKCqHkvyXuAKYB/ggqq6ZZRjkCT1jPoUEFV1OXD5\nqLe7m/aqU1JjZi36rEWfteibN7UY6UVgSdLew4+CkKSOMgAkqaMMAEnqqJFfBN7bJVlC7/bU1wHP\nBX4M3AxcBvxlVf18jMMbqSSvAX6LXi0OY+dafLaqHhnj8EbKn4u+JPsBb2aKWnTtrr75/nPhReAB\nSf6C3kdTfAXYBDwI7Ae8EHgD8CpgXVX9zdgGOSJJ/hL4e+BSpq7FvwQ+XlUL/n0c/lz0JfkQvV/+\nXweu4x/XYj/gA1V107jGOCoL4efCABiQ5Fer6uZfsHxf4HlVtXmEwxqLJIdU1S98O/swfRYCfy76\nkpxYVdN+wmKSZ9OrxaYRDmssFsLPhQEwg678kvtFkhwEPF5V3x/3WCTNHS8CD0hyfJK7kvxtklcm\nuQW4JsmWJMeMe3yjlOS5SS5M8gi9D7a6Ocn3kvynJE8d9/hGKcn1Sf44yfPHPZZxS7IiyVVJPptk\naZKNSR5Jcm2SV457fKOU5BlJPpzkllaDySRXJ/ntcY9tWAbAzv4zcALwB8BfA2uq6vnAm4D/Os6B\njcFn6X1UxwHAycAXgZfQu3Hgk+Mc2BgcBBwIXJXkG0l+P8lzxz2oMTkH+C/0LnL+HfDn7WdkXVvW\nJZ8D7gSOBT4EnA28A3hDkj8Z58CG5SmgAUmur6oj2/S9VbV0YNmNVXXE+EY3Wkm+WVWvGJi/rqpe\n1aa/XVUvHt/oRmuXn4vXAW8H3grcBlzUPsG2E5LcUFWvbNPfq6rnTbWsC6b4P3JtVf1akqcAt86H\n/yMeAezs4STvTvIHwPb2l97iJKuBH457cCM2meS32v6/D7gbIEno8M9NVf3vqvp39O7++BjwmjEP\nadR+kmRlkpOBSnISQJJ/Djw+3qGN3I+S/DpAkrcA2wDarZ8Z58CG5fsAdrYa+GPg58BKen/pXQHc\nA/zbMY5rHH4H+FN6h/Y3Ajse2nMw8MFxDWpMvrNrQ1U9Dny1fXXJ79I7BfRzeqc+3pPk0/Se69G1\n/yO/C3wqyXLgFnr/Z0gywTw5TeopIEnqqM4eyktS1xkAktRRBoAkdZQBMIQkq5IcPe5x7A2sRV97\nU1RX3w+wE2vRN59q4V1Awzka+KdJFlXV8eMezJhZi773AS9P8p2q+o1xD2bMrEXfvKmFdwFJuynJ\nM6vqB+Mex97AWvTNh1p4CmhISd407jHsLbpYiyTPSfKcNj2R5K1JXgawt/8nn2vWom++18IAGN75\n4x7AXqRTtUjybuD/AlcneQ+9z38/EfhSkjVjHdyIWYu+hVALTwENSDLdw00CvLGq9h/leMbJWvQl\n+Ra9ax9Pp/eu8BdU1f3tY7Kv6thnRFmLZiHUwovAO3sdvUcg7vq5PwGOGv1wxspa9P2sqh4FHk3y\n3aq6H6Cqtifp2l9Q1qJv3tfCANjZ1cCjVfW/dl2Q5PYxjGecrEVfJXlqVf2M3iE+8MSzcbt2GtVa\n9M37WngKSJpBkucBf19Vj+3Svhh4SVX99XhGNnrWom8h1MIAkKSOmheHKXuDdsFHWItB1qLPWvTN\nl1p4DWBAkrdOtwh4zijHMm7Wos9a9FmLvoVQCwNgZ5+n95zPqc6L7TfisYybteizFn3Wom/e18Jr\nAAOSXAesrqqbp1i20zOCFzpr0Wct+qxF30KohdcAdvZ+4PvTLPtXoxzIXsBa9FmLPmvRN+9r4RGA\nJHWU1wAGJFkErKGX3js+z3srcClwfnvDRydYiz5r0Wct+hZCLTwCGJDkIuBhYD2wpTUvAVYDB+/t\nn+09l6xFn7XosxZ9C6EWBsCA9gCHF8522UJkLfqsRZ+16FsItfAi8M62JTk5yRN1SfKUJL8BbB/j\nuMbBWvRZiz5r0Tfva2EA7OxU4G3AA0m+k+Q7wP3AW9uyLrEWfdaiz1r0zftaeApoGkmeBVBVD417\nLONmLfqsRZ+16JuvtfAIYBdJ/kmS51fVQ4P/mElePs5xjYO16LMWfdaib77XwgAYkOQU4NvAF5Pc\nkuTXBhZ/ejyjGg9r0Wct+qxF30KohQGwsz8CXtUe5fYu4DNJdryjL+Mb1lhYiz5r0Wct+uZ9LXwj\n2M72qar7AKrqG0neAHwlyVKm/sCnhcxa9FmLPmvRN+9r4RHAzn6Q5Pk7Zto/7uuBVcDLxjWoMbEW\nfdaiz1r0zftaeASws/ewSyhW1Q+SHAecMp4hjY216LMWfdaib97XwttAByRJzVCQYfosBNaiz1r0\nWYu+hVALTwHt7Kok70vvYc9PSLJvkjcmWU/vcz66wFr0WYs+a9E372vhEcCAJPsBvwP8JnA4vQ96\n2g/YB/gr4JyqumF8Ixwda9FnLfqsRd9CqIUBMI0kTwUOAX5cVQ+PezzjZC36rEWfteibr7UwACSp\no7wGIEkdZQBIUkcZAFqQkjye5Mb2GS3fTPKBDHxu+zTrLEvyr/fmbUlzyQDQQvXjqjqiql4GvAk4\nHjh9hnWWAU/ml/IotyXNGS8Ca0FK8sOqesbA/K8A19K7U+OXgc8A+7fF762qv0tyNfAS4C56z3k9\nGziT3tv7nwZ8sqr+fJzbkuaSAaAFaddfyq3tYeBFwA+An1fVT5IsBy6qqhVJXg/8h6p6c+u/Fnh2\nVX00ydOA/wOcXFV3jWtb0lzys4DURU8F/izJEcDjwHQP714JvDzJ29r8AcByen+1743bkmbFAFAn\ntNMyjwMP0js//wDwCnrXwX4y3WrA+6rqir11W9Lu8CKwFrwkE8B/B/6sfTDXAcB9VfVz4B303roP\nvdM1zxxY9QrgPe1dniR5YZL92/S3R7UtaU/xCEAL1dOT3EjvFMxj9C7EfrwtO4feY/zeCXwV+FFr\nvwl4PMk36T3S7xP07ta5PkmASeCkJIew8xOf9ti25qIQ0nS8CCzNUpI3A79SVWePeyzS7jAAJKmj\nvAYgSR1lAEhSRxkAktRRBoAkdZQBIEkdZQBIUkcZAJLUUf8fBPwHKHusb+UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1178574e0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(min(spray['Date']), max(spray['Date']))\n",
    "spray['Date'].groupby([spray[\"Date\"].dt.year, spray[\"Date\"].dt.month] ).count().plot(kind=\"bar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Get labels\n",
    "labels = train.WnvPresent.values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split station 1 and 2 and join horizontally\n",
    "weather_stn1 = weather[weather['Station']==1]\n",
    "weather_stn2 = weather[weather['Station']==2]\n",
    "weather_stn1 = weather_stn1.drop('Station', axis=1)\n",
    "weather_stn2 = weather_stn2.drop('Station', axis=1)\n",
    "weather = weather_stn1.merge(weather_stn2, on='Date')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "# Functions to extract month and day from dataset\n",
    "# You can also use parse_dates of Pandas.\n",
    "def create_month(x):\n",
    "    return x.split('-')[1]\n",
    "\n",
    "def create_day(x):\n",
    "    return x.split('-')[2]\n",
    "\n",
    "train['month'] = train.Date.apply(create_month)\n",
    "train['day'] = train.Date.apply(create_day)\n",
    "test['month'] = test.Date.apply(create_month)\n",
    "test['day'] = test.Date.apply(create_day)\n",
    "\n",
    "# Add integer latitude/longitude columns\n",
    "train['Lat_int'] = train.Latitude.apply(int)\n",
    "train['Long_int'] = train.Longitude.apply(int)\n",
    "test['Lat_int'] = test.Latitude.apply(int)\n",
    "test['Long_int'] = test.Longitude.apply(int)\n",
    "\n",
    "# drop address columns\n",
    "train = train.drop(['Address', 'AddressNumberAndStreet','WnvPresent', 'NumMosquitos'], axis = 1)\n",
    "test = test.drop(['Id', 'Address', 'AddressNumberAndStreet'], axis = 1)\n",
    "\n",
    "# Merge with weather data\n",
    "train = train.merge(weather, on='Date')\n",
    "test = test.merge(weather, on='Date')\n",
    "train = train.drop(['Date'], axis = 1)\n",
    "test = test.drop(['Date'], axis = 1)\n",
    "\n",
    "# Convert categorical data to numbers\n",
    "lbl = preprocessing.LabelEncoder()\n",
    "lbl.fit(list(train['Species'].values) + list(test['Species'].values))\n",
    "train['Species'] = lbl.transform(train['Species'].values)\n",
    "test['Species'] = lbl.transform(test['Species'].values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "lbl.fit(list(train['Street'].values) + list(test['Street'].values))\n",
    "train['Street'] = lbl.transform(train['Street'].values)\n",
    "test['Street'] = lbl.transform(test['Street'].values)\n",
    "\n",
    "lbl.fit(list(train['Trap'].values) + list(test['Trap'].values))\n",
    "train['Trap'] = lbl.transform(train['Trap'].values)\n",
    "test['Trap'] = lbl.transform(test['Trap'].values)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop columns with -1s\n",
    "train = train.ix[:,(train != -1).any(axis=0)]\n",
    "test = test.ix[:,(test != -1).any(axis=0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 1.2 Exploratory Data Analysis \n",
    "### 1.2.1 Histogram of continuous variables\n",
    "First, plot the histogram of each continous variable (SchedElapsedTime, Distance, ArrDelay and DepDelay) by 'Canceled'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Findings:\n",
    "* SchedElapsedTime and Distance both have a distribution skewed to the right. \n",
    "* Arrival and departure delay only have values when flights were not canceled (Cancel=0). So we should not include them as features.\n",
    "* 'SchedElapsedTime'and 'Distance' are highly correlated (correlation = 0.98). I'll choose only Distance as a feature in predition model to avoild multicollinearity in model. \n",
    "\n",
    "### 1.2.2 Frequency table of categorical features by Canceled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Findings:\n",
    "* No obvious increasing or decreasing trend when using Month or Hour as numeric variable, so it is reasonable to use them as categorical variable\n",
    "* Very few flights at 12AM, 1AM, 5AM. No flights at 2AM, 3AM, 4AM at all. Most flights were during day time.\n",
    "\n",
    "### 1.3 Feature Generation - One Hot Encoding\n",
    "use OneHotEncoder, LabelEncoder from sklearn to convert categorical variables (UniqueCarrier, Month and Hour) into binary variables to use for model fitting.\n",
    "\n",
    "* Build a custom transformer that will transform data to features we want:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# (base.BaseEstimator, base.TransformerMixin)-> this class will have .fit(),.fit_transform( mothods\n",
    "\n",
    "class ColumnSelectTransformer(base.BaseEstimator, base.TransformerMixin):\n",
    "    \n",
    "    def __init__(self, categorical_variables, numeric_variables): \n",
    "        # parameters taken by the transformer: a list of categorical_variables, a list of numeric_variables\n",
    "        self.categorical_variables = categorical_variables  # We will need these in transform()\n",
    "        self.numeric_variables     = numeric_variables\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        # Create empty list to store encode fit\n",
    "        self.label_encode = [] \n",
    "        self.OneHot_encode = []\n",
    "        \n",
    "        # fit LabelEncoder and OneHotEncoder to data\n",
    "        for i, column in enumerate(self.categorical_variables):\n",
    "            LE = LabelEncoder()                 # encode categorical variables as numeric\n",
    "            OHE = OneHotEncoder( sparse=False)  # do one hot encoding\n",
    "            OHE.fit(LE.fit_transform( X[column]).reshape(-1,1))\n",
    "            self.label_encode.append( LE )    \n",
    "            self.OneHot_encode.append( OHE )  \n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        X = X.reset_index() \n",
    "        df_cat = []  # empty list to store transformation result for each categorical variable\n",
    "        \n",
    "        # use LabelEncoder and OneHotEncoder to transform data\n",
    "        for i, column in enumerate(self.categorical_variables):\n",
    "            df_feature = self.OneHot_encode[i].transform(self.label_encode[i].transform( X[column]).reshape(-1,1))\n",
    "            df_feature = pd.DataFrame(df_feature)\n",
    "            df_feature.columns = [column  +'_'+str(x) for x in list(self.label_encode[i].classes_)]\n",
    "            df_cat.append(df_feature)\n",
    "            \n",
    "        # Merge these dummary variables with numeric features: \n",
    "        df_numeric = X[self.numeric_variables]\n",
    "        X = pd.concat([df_numeric]+ df_cat, axis=1) \n",
    "        \n",
    "        return X "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Transform data, \n",
    "* list feature names\n",
    "* look at the transformed feature matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "CT = ColumnSelectTransformer(categorical_variables=['UniqueCarrier','Month', 'Hour'], \n",
    "                                numeric_variables=['Distance'])\n",
    "df_feature = CT.fit_transform(data)\n",
    "columns = list(df_feature.columns)\n",
    "print(columns)\n",
    "df_feature.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Build a model to predict whether a flight will be canceled.\n",
    "Build prediction models using features generated above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier, ExtraTreesClassifier\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import roc_curve, auc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Split data into training set and test set\n",
    "Split data into traing set(N=5000) and test set(N=1000): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X =  data.drop('Canceled', axis=1)\n",
    "y = data['Canceled']\n",
    "X_train, X_test, y_train, y_test = train_test_split( X, y, test_size= 1000, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Fit different classifiers to training data and plot ROC curve using test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#max_depth= 10, n_estimators = 10, max_features= 10\n",
    "# name -> (line format, classifier)\n",
    "from sklearn.metrics import f1_score, confusion_matrix, accuracy_score\n",
    "\n",
    "CLASS_MAP ={\n",
    "    'Logistic Regression': ('-', LogisticRegression() ),\n",
    "    'Regularized Logistic Regression': ('.', GridSearchCV(LogisticRegression(penalty= 'l2'), {'C':np.logspace(-2, 2.0, num=10)}) ),\n",
    "    'Naive Bayes' :('--', GaussianNB() ),\n",
    "    'Decision Tree' :('.-',DecisionTreeClassifier(min_samples_split=10) ) ,\n",
    "    'Random Forest': (':', RandomForestClassifier(min_samples_split=20)),\n",
    "}\n",
    "\n",
    "plt.figure(figsize= (12,8))\n",
    "for name, (line_fmt, clf) in CLASS_MAP.items():\n",
    "    # train model\n",
    "    model = Pipeline([\n",
    "        ('ColSelect', CT ),# ColumnSelectTransformer\n",
    "        ('clf', clf)       # classifier\n",
    "    ])\n",
    "    \n",
    "    model.fit( X_train, y_train)\n",
    "    # predict probability on test data\n",
    "    preds = model.predict_proba(X_test)\n",
    "    pred = pd.Series(preds[:, 1])\n",
    "    \n",
    "    # Calcualte FPR, TPR for plotting ROC curve\n",
    "    fpr, tpr, thresholds = roc_curve(y_test, pred)\n",
    "    auc_score = auc(fpr, tpr)\n",
    "    label = '%s: AUC=%f' %(name, auc_score)\n",
    "    plt.plot(fpr, tpr, line_fmt, linewidth=5, label= label)\n",
    "    \n",
    "    train_pred = model.predict(X_train)\n",
    "    test_pred = model.predict(X_test)\n",
    "\n",
    "    print(name + ': Train Accuracy', accuracy_score(y_train, train_pred) , #sum(train_pred== y_train)/len(y_train),\n",
    "          ', Test Accuracy', accuracy_score(y_test, test_pred))\n",
    "\n",
    "    #print('Confusion matrix of train data:')\n",
    "    #print(confusion_matrix(y_train, train_pred))\n",
    "    print('Confusion matrix of test data:')\n",
    "    print(confusion_matrix(y_test, test_pred))\n",
    "plt.legend(loc= 'lower right')\n",
    "plt.xlabel('False positive rate')\n",
    "plt.ylabel('True positive rate')\n",
    "plt.title('ROC Curve of Different Classfiers')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#accuracy of predicting all flights to be not canceled:\n",
    "sum(y_test==0)/len(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Random Forest Classifier \n",
    "clf = ensemble.RandomForestClassifier(n_jobs=-1, n_estimators=1000, min_samples_split=1)\n",
    "clf.fit(train, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create predictions and submission file\n",
    "predictions = clf.predict_proba(test)[:,1]\n",
    "sample['WnvPresent'] = predictions\n",
    "sample.to_csv('beat_the_benchmark.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Findings: \n",
    "* All models have similar AUC. \n",
    "* All accuracy are similar to just predicting all flights not canceled(0.819). \n",
    "    * Random ForestTest Accuracy 0.816\n",
    "    * Decision TreeTest Accuracy 0.81\n",
    "    * Logistic RegressionTest Accuracy 0.819\n",
    "    * Regularized Logistic RegressionTest Accuracy 0.819\n",
    "    * Naive BayesTest Accuracy 0.547\n",
    "* Looking at the confusion matrix, we can see that logistic regression is actually predicting Canceled =0 for all, which is not a good choice.\n",
    "* I choose random forest classifier to be the final prediction model. This final model is very good but that's the best model using current features. \n",
    "\n",
    "### 2.3 Fit the final model using all data:\n",
    "Build a pipeline to do automatic feature transformation and model fitting.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "CT = ColumnSelectTransformer(categorical_variables=['UniqueCarrier','Month', 'Hour'], \n",
    "                                numeric_variables=['Distance'])\n",
    "#clf = LogisticRegression()\n",
    "clf = RandomForestClassifier(min_samples_split=20)\n",
    "pipe = Pipeline([\n",
    "        ('ColSelect', CT ),# ColumnSelectTransformer\n",
    "        ('clf', clf)       # classifier\n",
    "    ])\n",
    "\n",
    "pipe.fit(data, data['Canceled'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pipe.named_steps['clf'].feature_importances_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Examine the variance importance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "importance = pipe.named_steps['clf'].feature_importances_\n",
    "importance = pd.DataFrame(importance, index= columns, columns=[\"Importance\"])\n",
    "\n",
    "importance[\"Std\"] = np.std([tree.feature_importances_\n",
    "                            for tree in clf.estimators_], axis=0)\n",
    "\n",
    "importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_ = range(importance.shape[0])\n",
    "y_ = importance.ix[:, 0]\n",
    "yerr = importance.ix[:, 1]\n",
    "\n",
    "plt.bar(x_, y_, yerr=yerr, align=\"center\")\n",
    "plt.title('Variable Importance Plot')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpretation:\n",
    "Distance is the most important feature in the final model to predict flight cancellation.\n",
    "\n",
    "### Interpretation: \n",
    "*  The logistic regression coefficients give the change in the log odds of the outcome for a one unit increase in the predictor variable.\n",
    "* can get odds-ratio by exponentiating the coefficients.\n",
    "* Positive coefficient (odds ratio >1) suggest the odds of canceled is higher if that feature =0 compared with feature =1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Use the model output to predict whether a future flight will be canceled.\n",
    "* Ramdomly sample 10 rows from data to use as test data and show how prediction can be done for future new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}