Enron_final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Machine Learning to identify Enron fraudsters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Enron Corporation was an American energy, commodities, and services company based in Houston, Texas. It was founded in 1985 as the result of a merger between Houston Natural Gas and InterNorth, both relatively small regional companies. Before its bankruptcy on December 2, 2001, Enron employed approximately 20,000 staff and was one of the world's major electricity, natural gas, communications and pulp and paper companies. Fortune named Enron \"America's Most Innovative Company\" for six consecutive years. At the end of 2001, it was revealed that its reported financial condition was sustained by institutionalized, systematic, and creatively planned accounting fraud, known since as the Enron scandal. Enron has since become a well-known example of willful corporate fraud and corruption.\n",
    "\n",
    "In this project, I will build a model for identifying potential fraudsters based on financial and e-mail data. For this, the following steps will be performed: \n",
    "- data exploration (learning about the data, cleaning and preparing the data)\n",
    "- feature selection and engineering (selecting the most significant features and creating new ones) \n",
    "- reducing the dimensionality of the data using principal component analysis\n",
    "- selection and tuning a supervised machine learning algorithms \n",
    "- validating the algorithm to ensure acceptable performance of the model\n",
    "\n",
    "The ENRON Email dataset was collected and prepared by the CALO Project (A Cognitive Assistant that Learns and Organizes), see https://www.cs.cmu.edu/~./enron/ for details. The financial data was published in Payments to Insiders report by FindLaw and available at www.findlaw.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the Dataset and Question\n",
    "### Data exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with importing the data, transforming it into a dataframe, and displaying the header: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>to_messages</th>\n",
       "      <th>deferral_payments</th>\n",
       "      <th>total_payments</th>\n",
       "      <th>exercised_stock_options</th>\n",
       "      <th>bonus</th>\n",
       "      <th>restricted_stock</th>\n",
       "      <th>shared_receipt_with_poi</th>\n",
       "      <th>restricted_stock_deferred</th>\n",
       "      <th>total_stock_value</th>\n",
       "      <th>...</th>\n",
       "      <th>loan_advances</th>\n",
       "      <th>from_messages</th>\n",
       "      <th>other</th>\n",
       "      <th>from_this_person_to_poi</th>\n",
       "      <th>poi</th>\n",
       "      <th>director_fees</th>\n",
       "      <th>deferred_income</th>\n",
       "      <th>long_term_incentive</th>\n",
       "      <th>email_address</th>\n",
       "      <th>from_poi_to_this_person</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ALLEN PHILLIP K</th>\n",
       "      <td>201955</td>\n",
       "      <td>2902</td>\n",
       "      <td>2869717</td>\n",
       "      <td>4484442</td>\n",
       "      <td>1729541</td>\n",
       "      <td>4175000</td>\n",
       "      <td>126027</td>\n",
       "      <td>1407</td>\n",
       "      <td>-126027</td>\n",
       "      <td>1729541</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2195</td>\n",
       "      <td>152</td>\n",
       "      <td>65</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-3081055</td>\n",
       "      <td>304805</td>\n",
       "      <td>phillip.allen@enron.com</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BADUM JAMES P</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>178980</td>\n",
       "      <td>182466</td>\n",
       "      <td>257817</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>257817</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BANNANTINE JAMES M</th>\n",
       "      <td>477</td>\n",
       "      <td>566</td>\n",
       "      <td>NaN</td>\n",
       "      <td>916197</td>\n",
       "      <td>4046157</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1757552</td>\n",
       "      <td>465</td>\n",
       "      <td>-560222</td>\n",
       "      <td>5243487</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>29</td>\n",
       "      <td>864523</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-5104</td>\n",
       "      <td>NaN</td>\n",
       "      <td>james.bannantine@enron.com</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BAXTER JOHN C</th>\n",
       "      <td>267102</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1295738</td>\n",
       "      <td>5634343</td>\n",
       "      <td>6680544</td>\n",
       "      <td>1200000</td>\n",
       "      <td>3942714</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>10623258</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2660303</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1386055</td>\n",
       "      <td>1586055</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BAY FRANKLIN R</th>\n",
       "      <td>239671</td>\n",
       "      <td>NaN</td>\n",
       "      <td>260455</td>\n",
       "      <td>827696</td>\n",
       "      <td>NaN</td>\n",
       "      <td>400000</td>\n",
       "      <td>145796</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-82782</td>\n",
       "      <td>63014</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>69</td>\n",
       "      <td>NaN</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-201641</td>\n",
       "      <td>NaN</td>\n",
       "      <td>frank.bay@enron.com</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    salary to_messages deferral_payments total_payments  \\\n",
       "ALLEN PHILLIP K     201955        2902           2869717        4484442   \n",
       "BADUM JAMES P          NaN         NaN            178980         182466   \n",
       "BANNANTINE JAMES M     477         566               NaN         916197   \n",
       "BAXTER JOHN C       267102         NaN           1295738        5634343   \n",
       "BAY FRANKLIN R      239671         NaN            260455         827696   \n",
       "\n",
       "                   exercised_stock_options    bonus restricted_stock  \\\n",
       "ALLEN PHILLIP K                    1729541  4175000           126027   \n",
       "BADUM JAMES P                       257817      NaN              NaN   \n",
       "BANNANTINE JAMES M                 4046157      NaN          1757552   \n",
       "BAXTER JOHN C                      6680544  1200000          3942714   \n",
       "BAY FRANKLIN R                         NaN   400000           145796   \n",
       "\n",
       "                   shared_receipt_with_poi restricted_stock_deferred  \\\n",
       "ALLEN PHILLIP K                       1407                   -126027   \n",
       "BADUM JAMES P                          NaN                       NaN   \n",
       "BANNANTINE JAMES M                     465                   -560222   \n",
       "BAXTER JOHN C                          NaN                       NaN   \n",
       "BAY FRANKLIN R                         NaN                    -82782   \n",
       "\n",
       "                   total_stock_value           ...           loan_advances  \\\n",
       "ALLEN PHILLIP K              1729541           ...                     NaN   \n",
       "BADUM JAMES P                 257817           ...                     NaN   \n",
       "BANNANTINE JAMES M           5243487           ...                     NaN   \n",
       "BAXTER JOHN C               10623258           ...                     NaN   \n",
       "BAY FRANKLIN R                 63014           ...                     NaN   \n",
       "\n",
       "                   from_messages    other from_this_person_to_poi    poi  \\\n",
       "ALLEN PHILLIP K             2195      152                      65  False   \n",
       "BADUM JAMES P                NaN      NaN                     NaN  False   \n",
       "BANNANTINE JAMES M            29   864523                       0  False   \n",
       "BAXTER JOHN C                NaN  2660303                     NaN  False   \n",
       "BAY FRANKLIN R               NaN       69                     NaN  False   \n",
       "\n",
       "                    director_fees deferred_income long_term_incentive  \\\n",
       "ALLEN PHILLIP K               NaN        -3081055              304805   \n",
       "BADUM JAMES P                 NaN             NaN                 NaN   \n",
       "BANNANTINE JAMES M            NaN           -5104                 NaN   \n",
       "BAXTER JOHN C                 NaN        -1386055             1586055   \n",
       "BAY FRANKLIN R                NaN         -201641                 NaN   \n",
       "\n",
       "                                 email_address from_poi_to_this_person  \n",
       "ALLEN PHILLIP K        phillip.allen@enron.com                      47  \n",
       "BADUM JAMES P                              NaN                     NaN  \n",
       "BANNANTINE JAMES M  james.bannantine@enron.com                      39  \n",
       "BAXTER JOHN C                              NaN                     NaN  \n",
       "BAY FRANKLIN R             frank.bay@enron.com                     NaN  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#!/usr/bin/python\n",
    "\n",
    "import sys\n",
    "import pickle\n",
    "import pprint\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "sys.path.append(\"../tools/\")\n",
    "import tester\n",
    "from feature_format import featureFormat, targetFeatureSplit\n",
    "from tester import dump_classifier_and_data\n",
    "import tester\n",
    "\n",
    "### Load the dictionary containing the dataset\n",
    "with open(\"final_project_dataset.pkl\", \"r\") as data_file:\n",
    "    data_dict = pickle.load(data_file)\n",
    "\n",
    "# Transform dictionary to the Pandas DataFrame  \n",
    "df = pd.DataFrame.from_dict(data_dict, orient = 'index')\n",
    "# Names of the features \n",
    "features = list(df.keys())\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'NaN' values in the dataframe are actually text values, which should be replaces to proper NaN values. Let's make the replacement, look at the features, and analyse how many undefined values do we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 146 entries, ALLEN PHILLIP K to YEAP SOON\n",
      "Data columns (total 21 columns):\n",
      "salary                       95 non-null float64\n",
      "to_messages                  86 non-null float64\n",
      "deferral_payments            39 non-null float64\n",
      "total_payments               125 non-null float64\n",
      "exercised_stock_options      102 non-null float64\n",
      "bonus                        82 non-null float64\n",
      "restricted_stock             110 non-null float64\n",
      "shared_receipt_with_poi      86 non-null float64\n",
      "restricted_stock_deferred    18 non-null float64\n",
      "total_stock_value            126 non-null float64\n",
      "expenses                     95 non-null float64\n",
      "loan_advances                4 non-null float64\n",
      "from_messages                86 non-null float64\n",
      "other                        93 non-null float64\n",
      "from_this_person_to_poi      86 non-null float64\n",
      "poi                          146 non-null bool\n",
      "director_fees                17 non-null float64\n",
      "deferred_income              49 non-null float64\n",
      "long_term_incentive          66 non-null float64\n",
      "email_address                111 non-null object\n",
      "from_poi_to_this_person      86 non-null float64\n",
      "dtypes: bool(1), float64(19), object(1)\n",
      "memory usage: 24.1+ KB\n",
      "\n",
      "Total not-null data points: 1708\n",
      "Total NaN: 1358 (44.29%)\n"
     ]
    }
   ],
   "source": [
    "# Dataframe information\n",
    "df = df.replace(\"NaN\", np.nan)\n",
    "df.info()\n",
    "\n",
    "count = sum(df.count())\n",
    "count_nan = df.isnull().sum().sum()\n",
    "\n",
    "print \"\\nTotal not-null data points:\", count\n",
    "print 'Total NaN: {} ({:.2%})'.format(count_nan, float(count_nan)/float(count+count_nan))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also look at the allocation across classes: POI (person of interest) vs non-POI. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Allocation:\n",
      "non-POI    128\n",
      "POI         18\n",
      "Name: poi, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Allocation between classes\n",
    "poi_types = df.poi.value_counts()\n",
    "poi_types.index=['non-POI', 'POI']\n",
    "print \"Allocation:\\n\", poi_types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see: \n",
    "- There are 146 entries including 21 features (financial features, email features, and a binary classification \"poi\"). \n",
    "- Among the total of 3066 data points, values are not defined for 1358 (44.29%). We should certainly not be using loan_advances, director_fees, and restricted_stock_deferred for analysis and prediction since these features are not defined for the majority of entries. \n",
    "- Among 146 samples, there are 128 non-POI and 18 POI. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Outlier Investigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will analyse data for the presense of outliers and make a concious decision whether we should keep them in the dataset. We will use the definition from descriptive statistics (points higher than Q3 + 1.5IQR or lower than Q1 - 1.5IQR). Since Enron was a huge corporation with more than twenty thousands of employees, there were probably a lot of people not earning much. Thus, I will only look at the outliers over the upper fence. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/outliers.jpg\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                    outlier_columns_max\n",
      "FREVERT MARK A                       11\n",
      "TOTAL                                11\n",
      "LAY KENNETH L                        10\n",
      "LAVORATO JOHN J                      10\n",
      "WHALLEY LAWRENCE G                    9\n"
     ]
    }
   ],
   "source": [
    "# Upper fence value to detect outliers\n",
    "upper = df.quantile(.25) + 1.5 * (df.quantile(.75)-df.quantile(.25))\n",
    "\n",
    "# We should not use features such as email_address and poi for outliers analysis\n",
    "features_filt = features\n",
    "features_filt.remove('email_address')\n",
    "features_filt.remove('poi')\n",
    "\n",
    "# Finding outliers\n",
    "upper_o = pd.DataFrame((df[features_filt] > upper[features_filt]).sum(axis = 1), columns = ['outlier_columns_max']).\\\n",
    "    sort_values('outlier_columns_max', ascending = 0)  \n",
    "\n",
    "print upper_o.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Considering the first 5 of the samples with most number of outlier variables:\n",
    "1. Mark Frevert was named to a position in the chairman's office by Lay Kenneth, [according to Wikipedia](https://en.wikipedia.org/wiki/Enron_scandal). I consider that he represents a valid anomaly and therefore should be kept in the dataset. \n",
    "2. The 'TOTAL' should be definitely excluded as it is the total values of all the samples.  \n",
    "3. Kenneth Lay was a founder, chairman, and CEO, and should be kept in the dataset as a valid anomaly. \n",
    "4. John Lavorato was a chief executive of Enron Americas who received $5 mUSD as a retention bonus according to a [CNN article](http://edition.cnn.com/2002/LAW/02/09/enron.bonuses/). I do not think that he should be excluded. \n",
    "5. Lawrence 'Greg' Whalley was an Enron president and chief operating officer for a short period of time, and represents a valid anomaly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Exclude an outlier\n",
    "df = df.drop(['TOTAL'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, let's take a quick skim through on the indexes of our dataframe to detect anything unusual or incorrect. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['ALLEN PHILLIP K', 'BADUM JAMES P', 'BANNANTINE JAMES M',\n",
       "       'BAXTER JOHN C', 'BAY FRANKLIN R', 'BAZELIDES PHILIP J',\n",
       "       'BECK SALLY W', 'BELDEN TIMOTHY N', 'BELFER ROBERT',\n",
       "       'BERBERIAN DAVID', 'BERGSIEKER RICHARD P', 'BHATNAGAR SANJAY',\n",
       "       'BIBI PHILIPPE A', 'BLACHMAN JEREMY M', 'BLAKE JR. NORMAN P',\n",
       "       'BOWEN JR RAYMOND M', 'BROWN MICHAEL', 'BUCHANAN HAROLD G',\n",
       "       'BUTTS ROBERT H', 'BUY RICHARD B', 'CALGER CHRISTOPHER F',\n",
       "       'CARTER REBECCA C', 'CAUSEY RICHARD A', 'CHAN RONNIE',\n",
       "       'CHRISTODOULOU DIOMEDES', 'CLINE KENNETH W', 'COLWELL WESLEY',\n",
       "       'CORDES WILLIAM R', 'COX DAVID', 'CUMBERLAND MICHAEL S',\n",
       "       'DEFFNER JOSEPH M', 'DELAINEY DAVID W', 'DERRICK JR. JAMES V',\n",
       "       'DETMERING TIMOTHY J', 'DIETRICH JANET R', 'DIMICHELE RICHARD G',\n",
       "       'DODSON KEITH', 'DONAHUE JR JEFFREY M', 'DUNCAN JOHN H',\n",
       "       'DURAN WILLIAM D', 'ECHOLS JOHN B', 'ELLIOTT STEVEN',\n",
       "       'FALLON JAMES B', 'FASTOW ANDREW S', 'FITZGERALD JAY L',\n",
       "       'FOWLER PEGGY', 'FOY JOE', 'FREVERT MARK A', 'FUGH JOHN L',\n",
       "       'GAHN ROBERT S', 'GARLAND C KEVIN', 'GATHMANN WILLIAM D',\n",
       "       'GIBBS DANA R', 'GILLIS JOHN', 'GLISAN JR BEN F', 'GOLD JOSEPH',\n",
       "       'GRAMM WENDY L', 'GRAY RODNEY', 'HAEDICKE MARK E', 'HANNON KEVIN P',\n",
       "       'HAUG DAVID L', 'HAYES ROBERT E', 'HAYSLETT RODERICK J',\n",
       "       'HERMANN ROBERT J', 'HICKERSON GARY J', 'HIRKO JOSEPH',\n",
       "       'HORTON STANLEY C', 'HUGHES JAMES A', 'HUMPHREY GENE E',\n",
       "       'IZZO LAWRENCE L', 'JACKSON CHARLENE R', 'JAEDICKE ROBERT',\n",
       "       'KAMINSKI WINCENTY J', 'KEAN STEVEN J', 'KISHKILL JOSEPH G',\n",
       "       'KITCHEN LOUISE', 'KOENIG MARK E', 'KOPPER MICHAEL J',\n",
       "       'LAVORATO JOHN J', 'LAY KENNETH L', 'LEFF DANIEL P',\n",
       "       'LEMAISTRE CHARLES', 'LEWIS RICHARD', 'LINDHOLM TOD A',\n",
       "       'LOCKHART EUGENE E', 'LOWRY CHARLES P', 'MARTIN AMANDA K',\n",
       "       'MCCARTY DANNY J', 'MCCLELLAN GEORGE', 'MCCONNELL MICHAEL S',\n",
       "       'MCDONALD REBECCA', 'MCMAHON JEFFREY', 'MENDELSOHN JOHN',\n",
       "       'METTS MARK', 'MEYER JEROME J', 'MEYER ROCKFORD G',\n",
       "       'MORAN MICHAEL P', 'MORDAUNT KRISTINA M', 'MULLER MARK S',\n",
       "       'MURRAY JULIA H', 'NOLES JAMES L', 'OLSON CINDY K',\n",
       "       'OVERDYKE JR JERE C', 'PAI LOU L', 'PEREIRA PAULO V. FERRAZ',\n",
       "       'PICKERING MARK R', 'PIPER GREGORY F', 'PIRO JIM', 'POWERS WILLIAM',\n",
       "       'PRENTICE JAMES', 'REDMOND BRIAN L', 'REYNOLDS LAWRENCE',\n",
       "       'RICE KENNETH D', 'RIEKER PAULA H', 'SAVAGE FRANK',\n",
       "       'SCRIMSHAW MATTHEW', 'SHANKMAN JEFFREY A', 'SHAPIRO RICHARD S',\n",
       "       'SHARP VICTORIA T', 'SHELBY REX', 'SHERRICK JEFFREY B',\n",
       "       'SHERRIFF JOHN R', 'SKILLING JEFFREY K', 'STABLER FRANK',\n",
       "       'SULLIVAN-SHAKLOVITZ COLLEEN', 'SUNDE MARTIN', 'TAYLOR MITCHELL S',\n",
       "       'THE TRAVEL AGENCY IN THE PARK', 'THORN TERENCE H',\n",
       "       'TILNEY ELIZABETH A', 'UMANOFF ADAM S', 'URQUHART JOHN A',\n",
       "       'WAKEHAM JOHN', 'WALLS JR ROBERT H', 'WALTERS GARETH W',\n",
       "       'WASAFF GEORGE', 'WESTFAHL RICHARD K', 'WHALEY DAVID A',\n",
       "       'WHALLEY LAWRENCE G', 'WHITE JR THOMAS E', 'WINOKUR JR. HERBERT S',\n",
       "       'WODRASKA JOHN', 'WROBEL BRUCE', 'YEAGER F SCOTT', 'YEAP SOON'], dtype=object)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get indexes\n",
    "df.index.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The index 'THE TRAVEL AGENCY IN THE PARK' is definitely not a person - let's also exclude it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Exclude an outlier\n",
    "df = df.drop(['THE TRAVEL AGENCY IN THE PARK'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Optimize Feature Selection/Engineering\n",
    "### Create new features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, let's reiterate on available features and completeness of information for POI and non-POI. As analysed above, we will not be using loan_advances, director_fees, and restricted_stock_deferred features as they are too incomplete. We will drop them from the dataframe. Also, email_address identifies the person and is unique - we will discard this feature too. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = df.drop(['loan_advances','director_fees','restricted_stock_deferred', 'email_address'], 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, looking at the other features for POI and non-POI: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 18 entries, BELDEN TIMOTHY N to YEAGER F SCOTT\n",
      "Data columns (total 17 columns):\n",
      "salary                     17 non-null float64\n",
      "to_messages                14 non-null float64\n",
      "deferral_payments          5 non-null float64\n",
      "total_payments             18 non-null float64\n",
      "exercised_stock_options    12 non-null float64\n",
      "bonus                      16 non-null float64\n",
      "restricted_stock           17 non-null float64\n",
      "shared_receipt_with_poi    14 non-null float64\n",
      "total_stock_value          18 non-null float64\n",
      "expenses                   18 non-null float64\n",
      "from_messages              14 non-null float64\n",
      "other                      18 non-null float64\n",
      "from_this_person_to_poi    14 non-null float64\n",
      "poi                        18 non-null bool\n",
      "deferred_income            11 non-null float64\n",
      "long_term_incentive        12 non-null float64\n",
      "from_poi_to_this_person    14 non-null float64\n",
      "dtypes: bool(1), float64(16)\n",
      "memory usage: 2.4+ KB\n"
     ]
    }
   ],
   "source": [
    "df[(df.poi == True)].info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 126 entries, ALLEN PHILLIP K to YEAP SOON\n",
      "Data columns (total 17 columns):\n",
      "salary                     77 non-null float64\n",
      "to_messages                72 non-null float64\n",
      "deferral_payments          33 non-null float64\n",
      "total_payments             105 non-null float64\n",
      "exercised_stock_options    89 non-null float64\n",
      "bonus                      65 non-null float64\n",
      "restricted_stock           92 non-null float64\n",
      "shared_receipt_with_poi    72 non-null float64\n",
      "total_stock_value          107 non-null float64\n",
      "expenses                   76 non-null float64\n",
      "from_messages              72 non-null float64\n",
      "other                      73 non-null float64\n",
      "from_this_person_to_poi    72 non-null float64\n",
      "poi                        126 non-null bool\n",
      "deferred_income            37 non-null float64\n",
      "long_term_incentive        53 non-null float64\n",
      "from_poi_to_this_person    72 non-null float64\n",
      "dtypes: bool(1), float64(16)\n",
      "memory usage: 16.9+ KB\n"
     ]
    }
   ],
   "source": [
    "df[(df.poi == False)].info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks fine. Now, considering the relationships between features and relativity of their values, we will create a set of new ones. For instance, salary is a part of total_payments so the fraction of them intuitively feels like a better indicator for comparison betweeen employees. We will add the following features for further selection of the best ones (on the step 'intelligently select features'): \n",
    "- f_bonus (bonus/total_payments)\n",
    "- f_salary (salary/total_payments)\n",
    "- f_long_term_incentive (long_term_incentive/total_payments)\n",
    "- f_exercised_stock_options (exercised_stock_options/total_stock_value)\n",
    "- f_restricted_stock (restricted_stock/total_stock_value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df['f_bonus'] = df['bonus']/df['total_payments']\n",
    "df['f_salary'] = df['salary']/df['total_payments']\n",
    "df['f_long_term_incentive'] = df['long_term_incentive']/df['total_payments']\n",
    "df['f_exercised_stock_options'] = df['exercised_stock_options']/df['total_stock_value']\n",
    "df['f_restricted_stock'] = df['restricted_stock']/df['total_stock_value']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can create new indicators for e-mails: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df['f_from_poi'] = df['from_poi_to_this_person']/df['to_messages']\n",
    "df['f_to_poi'] = df['from_this_person_to_poi']/df['from_messages']\n",
    "df['f_shared_receipt_with_poi'] = df['shared_receipt_with_poi']/df['to_messages']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Properly scale features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have decided to apply scaling for the dataset. While scaling is not necessary for all of the machine learning algorithms in general, it would allow for greater flexibility moving further and applying different algorithms. We can use the following table for future reference (source - [stackexchange](https://stats.stackexchange.com/questions/244507/what-algorithms-need-feature-scaling-beside-from-svm)):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Algorithm           | Features need scaling   |\n",
    "|---------------------|-------------------------|\n",
    "| KNN                 | Yes                     |\n",
    "| Linear regression   | No (unless regularized) |\n",
    "| Logistic regression | No (unless regularized) |\n",
    "| Naive Bayes         | No                      |\n",
    "| Decision trees      | No                      |\n",
    "| Random forests      | No                      |\n",
    "| AdaBoost            | No                      |\n",
    "| Neural networks     | Yes                     |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Min-Max scaling will be used to scale the features. Since it does not work with NaN values, we will have to replace them with some values. I have not found any information on the Internet of whether missing values represent zeroes - thus, I will fill them with mean values and then scale: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>salary</th>\n",
       "      <th>to_messages</th>\n",
       "      <th>deferral_payments</th>\n",
       "      <th>total_payments</th>\n",
       "      <th>exercised_stock_options</th>\n",
       "      <th>bonus</th>\n",
       "      <th>restricted_stock</th>\n",
       "      <th>shared_receipt_with_poi</th>\n",
       "      <th>total_stock_value</th>\n",
       "      <th>expenses</th>\n",
       "      <th>...</th>\n",
       "      <th>long_term_incentive</th>\n",
       "      <th>from_poi_to_this_person</th>\n",
       "      <th>f_bonus</th>\n",
       "      <th>f_salary</th>\n",
       "      <th>f_long_term_incentive</th>\n",
       "      <th>f_exercised_stock_options</th>\n",
       "      <th>f_restricted_stock</th>\n",
       "      <th>f_from_poi</th>\n",
       "      <th>f_to_poi</th>\n",
       "      <th>f_shared_receipt_with_poi</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ALLEN PHILLIP K</th>\n",
       "      <td>0.181384</td>\n",
       "      <td>0.188510</td>\n",
       "      <td>0.455199</td>\n",
       "      <td>0.043302</td>\n",
       "      <td>0.050262</td>\n",
       "      <td>0.517654</td>\n",
       "      <td>0.157232</td>\n",
       "      <td>0.254575</td>\n",
       "      <td>0.036083</td>\n",
       "      <td>0.060014</td>\n",
       "      <td>...</td>\n",
       "      <td>0.046409</td>\n",
       "      <td>0.089015</td>\n",
       "      <td>0.168356</td>\n",
       "      <td>0.046984</td>\n",
       "      <td>0.006235</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.016593</td>\n",
       "      <td>0.074518</td>\n",
       "      <td>0.029613</td>\n",
       "      <td>0.474640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BADUM JAMES P</th>\n",
       "      <td>0.255325</td>\n",
       "      <td>0.133638</td>\n",
       "      <td>0.043109</td>\n",
       "      <td>0.001761</td>\n",
       "      <td>0.007411</td>\n",
       "      <td>0.142720</td>\n",
       "      <td>0.216047</td>\n",
       "      <td>0.212804</td>\n",
       "      <td>0.006142</td>\n",
       "      <td>0.014601</td>\n",
       "      <td>...</td>\n",
       "      <td>0.133420</td>\n",
       "      <td>0.122908</td>\n",
       "      <td>0.105040</td>\n",
       "      <td>0.243339</td>\n",
       "      <td>0.069352</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.153338</td>\n",
       "      <td>0.174670</td>\n",
       "      <td>0.184055</td>\n",
       "      <td>0.601038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BANNANTINE JAMES M</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.033726</td>\n",
       "      <td>0.144591</td>\n",
       "      <td>0.008846</td>\n",
       "      <td>0.117713</td>\n",
       "      <td>0.142720</td>\n",
       "      <td>0.251180</td>\n",
       "      <td>0.083892</td>\n",
       "      <td>0.107571</td>\n",
       "      <td>0.245623</td>\n",
       "      <td>...</td>\n",
       "      <td>0.133420</td>\n",
       "      <td>0.073864</td>\n",
       "      <td>0.105040</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.069352</td>\n",
       "      <td>0.787486</td>\n",
       "      <td>0.092007</td>\n",
       "      <td>0.317034</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.817260</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BAXTER JOHN C</th>\n",
       "      <td>0.240034</td>\n",
       "      <td>0.133638</td>\n",
       "      <td>0.214142</td>\n",
       "      <td>0.054405</td>\n",
       "      <td>0.194417</td>\n",
       "      <td>0.142497</td>\n",
       "      <td>0.377009</td>\n",
       "      <td>0.212804</td>\n",
       "      <td>0.217018</td>\n",
       "      <td>0.048343</td>\n",
       "      <td>...</td>\n",
       "      <td>0.298812</td>\n",
       "      <td>0.122908</td>\n",
       "      <td>0.028349</td>\n",
       "      <td>0.049487</td>\n",
       "      <td>0.044584</td>\n",
       "      <td>0.654594</td>\n",
       "      <td>0.102343</td>\n",
       "      <td>0.174670</td>\n",
       "      <td>0.184055</td>\n",
       "      <td>0.601038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BAY FRANKLIN R</th>\n",
       "      <td>0.215339</td>\n",
       "      <td>0.133638</td>\n",
       "      <td>0.055587</td>\n",
       "      <td>0.007991</td>\n",
       "      <td>0.086076</td>\n",
       "      <td>0.041614</td>\n",
       "      <td>0.158370</td>\n",
       "      <td>0.212804</td>\n",
       "      <td>0.002179</td>\n",
       "      <td>0.564241</td>\n",
       "      <td>...</td>\n",
       "      <td>0.133420</td>\n",
       "      <td>0.122908</td>\n",
       "      <td>0.081053</td>\n",
       "      <td>0.305082</td>\n",
       "      <td>0.069352</td>\n",
       "      <td>0.737972</td>\n",
       "      <td>0.660813</td>\n",
       "      <td>0.174670</td>\n",
       "      <td>0.184055</td>\n",
       "      <td>0.601038</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      salary  to_messages  deferral_payments  total_payments  \\\n",
       "ALLEN PHILLIP K     0.181384     0.188510           0.455199        0.043302   \n",
       "BADUM JAMES P       0.255325     0.133638           0.043109        0.001761   \n",
       "BANNANTINE JAMES M  0.000000     0.033726           0.144591        0.008846   \n",
       "BAXTER JOHN C       0.240034     0.133638           0.214142        0.054405   \n",
       "BAY FRANKLIN R      0.215339     0.133638           0.055587        0.007991   \n",
       "\n",
       "                    exercised_stock_options     bonus  restricted_stock  \\\n",
       "ALLEN PHILLIP K                    0.050262  0.517654          0.157232   \n",
       "BADUM JAMES P                      0.007411  0.142720          0.216047   \n",
       "BANNANTINE JAMES M                 0.117713  0.142720          0.251180   \n",
       "BAXTER JOHN C                      0.194417  0.142497          0.377009   \n",
       "BAY FRANKLIN R                     0.086076  0.041614          0.158370   \n",
       "\n",
       "                    shared_receipt_with_poi  total_stock_value  expenses  \\\n",
       "ALLEN PHILLIP K                    0.254575           0.036083  0.060014   \n",
       "BADUM JAMES P                      0.212804           0.006142  0.014601   \n",
       "BANNANTINE JAMES M                 0.083892           0.107571  0.245623   \n",
       "BAXTER JOHN C                      0.212804           0.217018  0.048343   \n",
       "BAY FRANKLIN R                     0.212804           0.002179  0.564241   \n",
       "\n",
       "                              ...              long_term_incentive  \\\n",
       "ALLEN PHILLIP K               ...                         0.046409   \n",
       "BADUM JAMES P                 ...                         0.133420   \n",
       "BANNANTINE JAMES M            ...                         0.133420   \n",
       "BAXTER JOHN C                 ...                         0.298812   \n",
       "BAY FRANKLIN R                ...                         0.133420   \n",
       "\n",
       "                    from_poi_to_this_person   f_bonus  f_salary  \\\n",
       "ALLEN PHILLIP K                    0.089015  0.168356  0.046984   \n",
       "BADUM JAMES P                      0.122908  0.105040  0.243339   \n",
       "BANNANTINE JAMES M                 0.073864  0.105040  0.000000   \n",
       "BAXTER JOHN C                      0.122908  0.028349  0.049487   \n",
       "BAY FRANKLIN R                     0.122908  0.081053  0.305082   \n",
       "\n",
       "                    f_long_term_incentive  f_exercised_stock_options  \\\n",
       "ALLEN PHILLIP K                  0.006235                   1.000000   \n",
       "BADUM JAMES P                    0.069352                   1.000000   \n",
       "BANNANTINE JAMES M               0.069352                   0.787486   \n",
       "BAXTER JOHN C                    0.044584                   0.654594   \n",
       "BAY FRANKLIN R                   0.069352                   0.737972   \n",
       "\n",
       "                    f_restricted_stock  f_from_poi  f_to_poi  \\\n",
       "ALLEN PHILLIP K               0.016593    0.074518  0.029613   \n",
       "BADUM JAMES P                 0.153338    0.174670  0.184055   \n",
       "BANNANTINE JAMES M            0.092007    0.317034  0.000000   \n",
       "BAXTER JOHN C                 0.102343    0.174670  0.184055   \n",
       "BAY FRANKLIN R                0.660813    0.174670  0.184055   \n",
       "\n",
       "                    f_shared_receipt_with_poi  \n",
       "ALLEN PHILLIP K                      0.474640  \n",
       "BADUM JAMES P                        0.601038  \n",
       "BANNANTINE JAMES M                   0.817260  \n",
       "BAXTER JOHN C                        0.601038  \n",
       "BAY FRANKLIN R                       0.601038  \n",
       "\n",
       "[5 rows x 25 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from pandas import DataFrame\n",
    "\n",
    "df = df.fillna(df.mean())\n",
    "\n",
    "# Preserving the dataframe structure\n",
    "i = df.index\n",
    "c = df.columns\n",
    "\n",
    "# Scaling\n",
    "scaler = MinMaxScaler()\n",
    "scaler.fit(df)\n",
    "\n",
    "df = DataFrame(scaler.transform(df), index=i, columns=c)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Intelligently select features "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Before moving forward, we need to transform data so it can be used with preprocessing functions which I used in the course of [Udacity Data Analyst Nanodegree](https://www.udacity.com/course/data-analyst-nanodegree--nd002). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Convert to a dictionary and use preprocessing functions. \n",
    "my_list = df.to_dict(orient='records')\n",
    "my_dataset = {}\n",
    "\n",
    "for index, values in zip(i, my_list):\n",
    "    my_dataset[index] = values\n",
    "\n",
    "# We will need a full list of features to select the best ones  \n",
    "features_list_full = my_dataset.itervalues().next().keys()\n",
    "\n",
    "# POI at the first position for further use with preprocessing functions \n",
    "features_list_full.remove('poi')\n",
    "features_used = list(features_list_full)\n",
    "features_list_full.insert(0, 'poi')\n",
    "\n",
    "# Use preprocessing functions \n",
    "data = featureFormat(my_dataset, features_list_full)\n",
    "labels, features = targetFeatureSplit(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's create and run a function which iterates over different samples and selects 10 most significant features using DecisionTreeClassifier and SelectKBest."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, let's think of all the available features. We have introduced some derived features above. Apparently, we will need to remove either original or derived features depending on their significance. I have written a helper function which can be used to do this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Function to remove derived or original values depending on their strength\n",
    "feature_singlets = {'f_bonus':'bonus', 'f_salary':'salary', \n",
    "                    'f_long_term_incentive':'long_term_incentive', \n",
    "                    'f_exercised_stock_options':'exercised_stock_options',\n",
    "                    'f_restricted_stock':'restricted_stock'\n",
    "                    }\n",
    "\n",
    "feature_pairs = {'f_from_poi':['from_poi_to_this_person', 'to_messages'], \n",
    "                 'f_to_poi':['from_this_person_to_poi', 'from_messages'], \n",
    "                 'f_shared_receipt_with_poi':['shared_receipt_with_poi', 'to_messages']\n",
    "                }\n",
    "\n",
    "def rm_excessive_features(features_source, feature_pairs, feature_singlets):\n",
    "    rm_values = set()\n",
    "    \n",
    "    for key, value in feature_pairs.iteritems(): \n",
    "        # Calculating original and derived weight \n",
    "        # Comparing by weighted average\n",
    "        w_org = (features_source[value[0]] + features_source[value[1]])/2\n",
    "        w_drv = features_source[key]\n",
    "        if w_drv >= w_org: \n",
    "            #print 'Removing feature:', value[0], value[1]\n",
    "            rm_values.add(value[0])\n",
    "            rm_values.add(value[1])                                  \n",
    "        else: \n",
    "            #print 'Removing feature:', key\n",
    "            rm_values.add(key)  \n",
    "    \n",
    "    for key, value in feature_singlets.iteritems(): \n",
    "        w_org = features_source[value] \n",
    "        w_drv = features_source[key]\n",
    "        if w_drv >= w_org: \n",
    "            #print 'Removing feature:', value\n",
    "            rm_values.add(value)                            \n",
    "        else: \n",
    "            #print 'Removing feature:', key\n",
    "            rm_values.add(key)     \n",
    "            \n",
    "    # Removing less significant features\n",
    "    for rm_val in rm_values: \n",
    "        features_source.pop(rm_val, None)    \n",
    "    \n",
    "    return features_source"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, I will write a finction to produce shuffled folds, detect the most significant features, and removing excessive features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Feature combination 1 ['f_bonus', 'deferral_payments', 'f_from_poi', 'f_salary', 'deferred_income', 'other', 'f_shared_receipt_with_poi', 'from_messages', 'expenses', 'from_this_person_to_poi', 'long_term_incentive', 'total_payments', 'f_exercised_stock_options', 'f_restricted_stock', 'total_stock_value']\n",
      "\n",
      "Feature combination 2 ['f_salary', 'deferral_payments', 'expenses', 'deferred_income', 'f_bonus', 'other', 'f_shared_receipt_with_poi', 'from_poi_to_this_person', 'from_messages', 'from_this_person_to_poi', 'long_term_incentive', 'f_exercised_stock_options', 'total_payments', 'f_restricted_stock', 'total_stock_value']\n",
      "\n",
      "Feature combination 3 ['to_messages', 'expenses', 'f_salary', 'deferral_payments', 'f_bonus', 'shared_receipt_with_poi', 'other', 'deferred_income', 'from_poi_to_this_person', 'long_term_incentive', 'from_messages', 'restricted_stock', 'from_this_person_to_poi', 'exercised_stock_options', 'total_payments', 'total_stock_value']\n",
      "\n",
      "Feature combination 4 ['deferral_payments', 'f_salary', 'deferred_income', 'other', 'f_bonus', 'expenses', 'f_shared_receipt_with_poi', 'from_poi_to_this_person', 'from_messages', 'from_this_person_to_poi', 'restricted_stock', 'long_term_incentive', 'total_payments', 'total_stock_value', 'f_exercised_stock_options']\n",
      "\n",
      "Feature combination 5 ['to_messages', 'f_salary', 'deferral_payments', 'deferred_income', 'f_bonus', 'expenses', 'from_poi_to_this_person', 'other', 'from_messages', 'f_exercised_stock_options', 'from_this_person_to_poi', 'shared_receipt_with_poi', 'restricted_stock', 'total_stock_value', 'total_payments', 'f_long_term_incentive']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import StratifiedShuffleSplit\n",
    "from sklearn import tree\n",
    "from sklearn.feature_selection import SelectKBest\n",
    "from sklearn.feature_selection import chi2\n",
    "# To sort the result\n",
    "import operator\n",
    "\n",
    "# Create a function to return combinations of the most significant features \n",
    "def select_features(features_used, features, labels): \n",
    "    # Create a dictionary to populate from feature_used array, and a results list\n",
    "\n",
    "    best_f_dict = dict.fromkeys(features_used, 0)\n",
    "    results = []\n",
    "\n",
    "    # Create a Stratified ShuffleSplit cross-validaton with 10 splits (folds)\n",
    "    # 10 is a default value, which should be enough for a dataset containing 144 values\n",
    "    cv = StratifiedShuffleSplit(labels, 10)\n",
    "    #print \"\\nFeature engineering:\"\n",
    "    \n",
    "    # Try classifiers on folds and select the best features\n",
    "    for train_idx, test_idx in cv:\n",
    "        features_train = []\n",
    "        features_test  = []\n",
    "        labels_train   = []\n",
    "        labels_test    = []\n",
    "\n",
    "        for ii in train_idx:\n",
    "            features_train.append( features[ii] )\n",
    "            labels_train.append( labels[ii] )\n",
    "        for jj in test_idx:\n",
    "            features_test.append( features[jj] )\n",
    "            labels_test.append( labels[jj] )\n",
    "\n",
    "        # Create a Decision Tree classifier \n",
    "        clf = tree.DecisionTreeClassifier()    \n",
    "        # Fit the classifier using training set, and test on test set\n",
    "        clf.fit(features_train, labels_train)\n",
    "        # Detect most important features from the tree\n",
    "        importances = clf.feature_importances_\n",
    "        indices = np.argsort(importances)[::-1]   \n",
    "        # Get 10 most important features of the Decision tree \n",
    "        for i in range(10):\n",
    "            best_f_val = features_used[indices[i]]\n",
    "            best_f_dict[best_f_val] = best_f_dict[best_f_val] + 1\n",
    "\n",
    "        # Detect most important features using SelectKBest\n",
    "        selector = SelectKBest(chi2, k=10)\n",
    "        selector.fit(features_train, labels_train)\n",
    "        features_train_transformed = selector.transform(features_train) \n",
    "        support = np.asarray(selector.get_support())\n",
    "\n",
    "        # Get an array with the most important features \n",
    "        features_arr = np.asarray(features_used)\n",
    "        columns_support = features_arr[support]\n",
    "        for val in columns_support: \n",
    "            best_f_dict[val] = best_f_dict[val] + 1\n",
    "\n",
    "    # Remove less significant features \n",
    "    best_f_dict = rm_excessive_features(best_f_dict, feature_pairs, feature_singlets)      \n",
    "            \n",
    "    # Get the list of sorted features sorted by their importance \n",
    "    best_f_list = sorted(best_f_dict.iteritems(), key=operator.itemgetter(1), reverse=True)\n",
    "    \n",
    "    #print \"\\nThe most important features by scores:\"\n",
    "    #pprint.pprint(best_f_list)\n",
    "\n",
    "    # Save the list of features for future reference, sorted by their importance\n",
    "    features_filtered = []\n",
    "    for val in best_f_list: \n",
    "        features_filtered.append(val[0])\n",
    "        \n",
    "    results = features_filtered \n",
    "    return results \n",
    "\n",
    "# Generate a list of sets of the most significant features to evaluate performance on different combinations \n",
    "features_filtered_set = []\n",
    "for i in range (1, 6):\n",
    "    best_set_found = select_features(features_used, features, labels)\n",
    "    print \"\\nFeature combination\", i, best_set_found\n",
    "    features_filtered_set.append(best_set_found)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the selected features further. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pick and Tune an Algorithm\n",
    "### Pick an algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I dediced to pick three algorithms: Decision Tree, Gaussian Naive Bayes, and K-nearest neighbours. First, I will build a few auxiliary functions and then implement Decision Tree classifier: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# Function which return shuffled testing and training sets \n",
    "def shuffle_split(labels, features): \n",
    "    features_train = []\n",
    "    features_test  = []\n",
    "    labels_train   = []\n",
    "    labels_test    = []   \n",
    "    # 10 is a default value, which should be enough for a dataset containing 144 values\n",
    "    cv = StratifiedShuffleSplit(labels, 10)\n",
    "    # Create leatures and labels shuffled dataset\n",
    "    for train_idx, test_idx in cv:    \n",
    "        for ii in train_idx:\n",
    "            features_train.append( features[ii] )\n",
    "            labels_train.append( labels[ii] )\n",
    "        for jj in test_idx:\n",
    "            features_test.append( features[jj] )\n",
    "            labels_test.append( labels[jj] )\n",
    "    return features_train, features_test, labels_train, labels_test\n",
    "\n",
    "# Function to produce shuffled sets for each of identified features combination and to fit / evaluate a classifier on each\n",
    "def prepare_evaluate(my_dataset, features_filtered_set, clf):\n",
    "    i = 1\n",
    "    for features_filtered in features_filtered_set:       \n",
    "        # Prepare features list for tester and for data split \n",
    "        features_tester = list(features_filtered)\n",
    "        features_tester.insert(0, 'poi')\n",
    "        data = featureFormat(my_dataset, features_tester)\n",
    "        labels, features = targetFeatureSplit(data)\n",
    "        # Get the shuffled sets\n",
    "        features_train, features_test, labels_train, labels_test = shuffle_split(labels, features)\n",
    "        print \"Evaluating classifier on the feature set\", i\n",
    "        # Fit the classifier\n",
    "        clf.fit(features_train, labels_train)\n",
    "        # Call tester \n",
    "        tester.test_classifier(clf, my_dataset, features_tester)\n",
    "        i += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating classifier on the feature set 1\n",
      "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
      "            max_features=None, max_leaf_nodes=None,\n",
      "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
      "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
      "            presort=False, random_state=None, splitter='best')\n",
      "\tAccuracy: 0.82160\tPrecision: 0.32631\tRecall: 0.31750\tF1: 0.32184\tF2: 0.31922\n",
      "\tTotal predictions: 15000\tTrue positives:  635\tFalse positives: 1311\tFalse negatives: 1365\tTrue negatives: 11689\n",
      "\n",
      "Evaluating classifier on the feature set 2\n",
      "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
      "            max_features=None, max_leaf_nodes=None,\n",
      "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
      "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
      "            presort=False, random_state=None, splitter='best')\n",
      "\tAccuracy: 0.81460\tPrecision: 0.30582\tRecall: 0.30750\tF1: 0.30666\tF2: 0.30716\n",
      "\tTotal predictions: 15000\tTrue positives:  615\tFalse positives: 1396\tFalse negatives: 1385\tTrue negatives: 11604\n",
      "\n",
      "Evaluating classifier on the feature set 3\n",
      "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
      "            max_features=None, max_leaf_nodes=None,\n",
      "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
      "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
      "            presort=False, random_state=None, splitter='best')\n",
      "\tAccuracy: 0.80533\tPrecision: 0.28178\tRecall: 0.29700\tF1: 0.28919\tF2: 0.29383\n",
      "\tTotal predictions: 15000\tTrue positives:  594\tFalse positives: 1514\tFalse negatives: 1406\tTrue negatives: 11486\n",
      "\n",
      "Evaluating classifier on the feature set 4\n",
      "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
      "            max_features=None, max_leaf_nodes=None,\n",
      "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
      "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
      "            presort=False, random_state=None, splitter='best')\n",
      "\tAccuracy: 0.81253\tPrecision: 0.29618\tRecall: 0.29500\tF1: 0.29559\tF2: 0.29524\n",
      "\tTotal predictions: 15000\tTrue positives:  590\tFalse positives: 1402\tFalse negatives: 1410\tTrue negatives: 11598\n",
      "\n",
      "Evaluating classifier on the feature set 5\n",
      "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
      "            max_features=None, max_leaf_nodes=None,\n",
      "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
      "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
      "            presort=False, random_state=None, splitter='best')\n",
      "\tAccuracy: 0.79760\tPrecision: 0.23271\tRecall: 0.22550\tF1: 0.22905\tF2: 0.22691\n",
      "\tTotal predictions: 15000\tTrue positives:  451\tFalse positives: 1487\tFalse negatives: 1549\tTrue negatives: 11513\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Create and use a Decision Tree classifier \n",
    "clf_tree = tree.DecisionTreeClassifier()\n",
    "prepare_evaluate(my_dataset, features_filtered_set, clf_tree)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GaussianNB:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating classifier on the feature set 1\n",
      "GaussianNB(priors=None)\n",
      "\tAccuracy: 0.83127\tPrecision: 0.32359\tRecall: 0.24350\tF1: 0.27789\tF2: 0.25618\n",
      "\tTotal predictions: 15000\tTrue positives:  487\tFalse positives: 1018\tFalse negatives: 1513\tTrue negatives: 11982\n",
      "\n",
      "Evaluating classifier on the feature set 2\n",
      "GaussianNB(priors=None)\n",
      "\tAccuracy: 0.83193\tPrecision: 0.32575\tRecall: 0.24350\tF1: 0.27868\tF2: 0.25645\n",
      "\tTotal predictions: 15000\tTrue positives:  487\tFalse positives: 1008\tFalse negatives: 1513\tTrue negatives: 11992\n",
      "\n",
      "Evaluating classifier on the feature set 3\n",
      "GaussianNB(priors=None)\n",
      "\tAccuracy: 0.82800\tPrecision: 0.31761\tRecall: 0.25250\tF1: 0.28134\tF2: 0.26330\n",
      "\tTotal predictions: 15000\tTrue positives:  505\tFalse positives: 1085\tFalse negatives: 1495\tTrue negatives: 11915\n",
      "\n",
      "Evaluating classifier on the feature set 4\n",
      "GaussianNB(priors=None)\n",
      "\tAccuracy: 0.82873\tPrecision: 0.31562\tRecall: 0.24350\tF1: 0.27491\tF2: 0.25516\n",
      "\tTotal predictions: 15000\tTrue positives:  487\tFalse positives: 1056\tFalse negatives: 1513\tTrue negatives: 11944\n",
      "\n",
      "Evaluating classifier on the feature set 5\n",
      "GaussianNB(priors=None)\n",
      "\tAccuracy: 0.83640\tPrecision: 0.33946\tRecall: 0.24000\tF1: 0.28120\tF2: 0.25494\n",
      "\tTotal predictions: 15000\tTrue positives:  480\tFalse positives:  934\tFalse negatives: 1520\tTrue negatives: 12066\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "# Create and use a GaussianNB classifier \n",
    "clf_gnb = GaussianNB()\n",
    "prepare_evaluate(my_dataset, features_filtered_set, clf_gnb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GaussianNB with feature selection and PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating classifier on the feature set 1\n",
      "Pipeline(steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=3, random_state=None,\n",
      "  svd_solver='auto', tol=0.0, whiten=False)), ('model', GaussianNB(priors=None))])\n",
      "\tAccuracy: 0.85093\tPrecision: 0.22685\tRecall: 0.04900\tF1: 0.08059\tF2: 0.05811\n",
      "\tTotal predictions: 15000\tTrue positives:   98\tFalse positives:  334\tFalse negatives: 1902\tTrue negatives: 12666\n",
      "\n",
      "Evaluating classifier on the feature set 2\n",
      "Pipeline(steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=3, random_state=None,\n",
      "  svd_solver='auto', tol=0.0, whiten=False)), ('model', GaussianNB(priors=None))])\n",
      "\tAccuracy: 0.86160\tPrecision: 0.39730\tRecall: 0.07350\tF1: 0.12405\tF2: 0.08781\n",
      "\tTotal predictions: 15000\tTrue positives:  147\tFalse positives:  223\tFalse negatives: 1853\tTrue negatives: 12777\n",
      "\n",
      "Evaluating classifier on the feature set 3\n",
      "Pipeline(steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=3, random_state=None,\n",
      "  svd_solver='auto', tol=0.0, whiten=False)), ('model', GaussianNB(priors=None))])\n",
      "\tAccuracy: 0.83860\tPrecision: 0.37062\tRecall: 0.30150\tF1: 0.33251\tF2: 0.31318\n",
      "\tTotal predictions: 15000\tTrue positives:  603\tFalse positives: 1024\tFalse negatives: 1397\tTrue negatives: 11976\n",
      "\n",
      "Evaluating classifier on the feature set 4\n",
      "Pipeline(steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=3, random_state=None,\n",
      "  svd_solver='auto', tol=0.0, whiten=False)), ('model', GaussianNB(priors=None))])\n",
      "\tAccuracy: 0.85833\tPrecision: 0.37769\tRecall: 0.09650\tF1: 0.15372\tF2: 0.11338\n",
      "\tTotal predictions: 15000\tTrue positives:  193\tFalse positives:  318\tFalse negatives: 1807\tTrue negatives: 12682\n",
      "\n",
      "Evaluating classifier on the feature set 5\n",
      "Pipeline(steps=[('pca', PCA(copy=True, iterated_power='auto', n_components=3, random_state=None,\n",
      "  svd_solver='auto', tol=0.0, whiten=False)), ('model', GaussianNB(priors=None))])\n",
      "\tAccuracy: 0.83247\tPrecision: 0.22028\tRecall: 0.10100\tF1: 0.13850\tF2: 0.11327\n",
      "\tTotal predictions: 15000\tTrue positives:  202\tFalse positives:  715\tFalse negatives: 1798\tTrue negatives: 12285\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Creating a pipeline with PCA and the GaussianNB\n",
    "pca = PCA(n_components=3)\n",
    "clf_gnb_n = GaussianNB()\n",
    "pipe = Pipeline([('pca', pca), ('model', clf_gnb_n)])\n",
    "\n",
    "prepare_evaluate(my_dataset, features_filtered_set, pipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "K-nearest neighbours:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating classifier on the feature set 1\n",
      "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
      "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
      "           weights='uniform')\n",
      "\tAccuracy: 0.86687\tPrecision: 1.00000\tRecall: 0.00150\tF1: 0.00300\tF2: 0.00187\n",
      "\tTotal predictions: 15000\tTrue positives:    3\tFalse positives:    0\tFalse negatives: 1997\tTrue negatives: 13000\n",
      "\n",
      "Evaluating classifier on the feature set 2\n",
      "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
      "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
      "           weights='uniform')\n",
      "\tAccuracy: 0.87267\tPrecision: 0.98913\tRecall: 0.04550\tF1: 0.08700\tF2: 0.05623\n",
      "\tTotal predictions: 15000\tTrue positives:   91\tFalse positives:    1\tFalse negatives: 1909\tTrue negatives: 12999\n",
      "\n",
      "Evaluating classifier on the feature set 3\n",
      "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
      "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
      "           weights='uniform')\n",
      "\tAccuracy: 0.86747\tPrecision: 0.53371\tRecall: 0.04750\tF1: 0.08724\tF2: 0.05808\n",
      "\tTotal predictions: 15000\tTrue positives:   95\tFalse positives:   83\tFalse negatives: 1905\tTrue negatives: 12917\n",
      "\n",
      "Evaluating classifier on the feature set 4\n",
      "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
      "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
      "           weights='uniform')\n",
      "\tAccuracy: 0.87153\tPrecision: 0.83486\tRecall: 0.04550\tF1: 0.08630\tF2: 0.05611\n",
      "\tTotal predictions: 15000\tTrue positives:   91\tFalse positives:   18\tFalse negatives: 1909\tTrue negatives: 12982\n",
      "\n",
      "Evaluating classifier on the feature set 5\n",
      "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
      "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
      "           weights='uniform')\n",
      "\tAccuracy: 0.86560\tPrecision: 0.47080\tRecall: 0.06450\tF1: 0.11346\tF2: 0.07796\n",
      "\tTotal predictions: 15000\tTrue positives:  129\tFalse positives:  145\tFalse negatives: 1871\tTrue negatives: 12855\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# Create and use a kNN classifier \n",
    "clf_knn = KNeighborsClassifier()\n",
    "prepare_evaluate(my_dataset, features_filtered_set, clf_knn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summary of the results for the first feature set is: \n",
    "\n",
    "| Algorithm                                   | Precision | Recall  | F1      |\n",
    "|---------------------------------------------|-----------|---------|---------|\n",
    "| Decision trees                              | 0.32631   | 0.31750 | 0.32184 |\n",
    "| GaussianNB                                  | 0.32359   | 0.24350 | 0.27789 |\n",
    "| GaussianNB (with PCA and feature selection) | 0.22685   | 0.04900 | 0.08059 |\n",
    "| K Nearest Neighbours                        | 1.00000   | 0.00150 | 0.00300 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For all the feature sets, comparing F1 score: \n",
    "\n",
    "| Algorithm                                   | F1: set 1 | F1: set 2 | F1: set 3 | F1: set 4 | F1: set 5 |\n",
    "|---------------------------------------------|-----------|-----------|-----------|-----------|-----------|\n",
    "| Decision trees                              | 0.32184   | 0.30666   | 0.28919   | 0.29559   | 0.22905   |\n",
    "| GaussianNB                                  | 0.27789   | 0.27868   | 0.28134   | 0.27491   | 0.28120   |\n",
    "| GaussianNB (with PCA and feature selection) | 0.08059   | 0.12405   | 0.33251   | 0.15372   | 0.13850   |\n",
    "| K Nearest Neighbours                        | 0.00300   | 0.08700   | 0.08724   | 0.08630   | 0.11346   |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decision trees looks to perform the best across different feature sets, giving the highest F1 score in average. Gaussian Naive Bayes with PCA and feature selection, as well as kNN, give quite poor results for F1. Looking at the precision value for kNN on the first feature set, seems that it is prone to overfitting in our case. Let's proceed with the decision tree and fine-tune the algorithm. I will proceed with the first feature set as it gives the best results for this algorithm. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "features_filtered = features_filtered_set[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A general complication in Machine Learning is that learning algorithms require us to set parameters before using the models, and suboptimal selection of parameters can significantly influence the performance (as we saw above with the GaussianNB classifier). Our goal should be to optimise the parameters that impact the model in order to enable the algorithm to perform the best (\"best\" can be defined with metrics such as recall, precision, and F1 score). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sklearn [GridSearchCV](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) is a way of systematically working through multiple combinations of parameter tunes, cross-validating as it goes to determine which tune gives the best performance. We can use it to tune the Decision tree classifier. I decided to select the following parameters for tuning:\n",
    "- criterion (the function to measure the quality of a split),\n",
    "- splitter (the strategy used to choose the split at each node), \n",
    "- min_samples_split (the minimum number of samples required to split an internal node), \n",
    "- min_samples_leaf (the minimum number of samples required to be at a leaf node). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters: {'min_samples_split': 2, 'splitter': 'best', 'min_samples_leaf': 1}\n"
     ]
    }
   ],
   "source": [
    "from sklearn import svm, grid_search\n",
    "\n",
    "# Parameters for fine tuning  \n",
    "parameters = {'splitter':('best','random'), \n",
    "    'min_samples_split':(2, 5, 10, 15), \n",
    "    'min_samples_leaf':(1, 3, 6, 8, 10)}\n",
    "\n",
    "# Update the dataset\n",
    "features_tester = list(features_filtered)\n",
    "features_tester.insert(0, 'poi')\n",
    "data = featureFormat(my_dataset, features_tester)\n",
    "labels, features = targetFeatureSplit(data)\n",
    "features_train, features_test, labels_train, labels_test = shuffle_split(labels, features)\n",
    "\n",
    "# Create and use a classifier for fitting\n",
    "clf_tree = tree.DecisionTreeClassifier()\n",
    "clf_tree = grid_search.GridSearchCV(clf_tree, parameters) \n",
    "\n",
    "clf_tree.fit(features_train, labels_train)\n",
    "\n",
    "print \"Best parameters:\", clf_tree.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
      "            max_features=None, max_leaf_nodes=None,\n",
      "            min_impurity_split=1e-07, min_samples_leaf=1,\n",
      "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
      "            presort=False, random_state=None, splitter='best')\n",
      "\tAccuracy: 0.81927\tPrecision: 0.31834\tRecall: 0.31150\tF1: 0.31489\tF2: 0.31285\n",
      "\tTotal predictions: 15000\tTrue positives:  623\tFalse positives: 1334\tFalse negatives: 1377\tTrue negatives: 11666\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Using the parameters\n",
    "clf_tree = tree.DecisionTreeClassifier(min_samples_split = 2,\n",
    "                             splitter = 'best',\n",
    "                             min_samples_leaf = 1)\n",
    "\n",
    "clf_tree.fit(features_train, labels_train)\n",
    "\n",
    "# Call tester \n",
    "tester.dump_classifier_and_data(clf_tree, my_dataset, features_tester)\n",
    "tester.main() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validate and evaluate \n",
    "### Validation and its importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most important notions of data science are related to overfitting and generalization. If we allow ourselves enough flexibility in searching for patterns in a particular dataset, we will find patterns. Generally, we are interested in patterns\n",
    "that generalize—that predict well for instances that we have not yet observed. If the patterns do not generalise, then we have created a model which is overfit. \n",
    "\n",
    "To avoid overfitting, validation is used to build and test a model. In this process, a trained model is evaluated with a testing subset, which is 'hold out' of the original dataset not used for training. The main purpose of using the testing data set is to test the generalisation ability of a trained model. \n",
    "\n",
    "However, creating training and testing sets by splitting without proper data shuffle could lead to uneven mix of data across classes, especially if data is skewed or grouped sequentally in the original dataset. Cross-validation is a proven approach to deal with this issue. Unlike splitting the data into one training and one holdout set, cross-validation computes its estimates over all the data by performing multiple splits and systematically swapping out samples for testing. Cross-validation begins by splitting a labeled dataset into k partitions called folds, and then then iterates training and testing k times when a different fold is chosen as the test data in each iteration of the cross-validation. \n",
    "\n",
    "In sklearn, [StratifiedShuffleSplit](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html) can be used to implement cross-validation. This has been included in the testing function (tester.py), which shuffles the data and splits it into 1000 folds. We have already used StratifiedShuffleSplit for sets preparation in order to select the most significant features, train different classifiers, and fine-tune the best performing one. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation Metrics and Algorithm Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several different ways to measure the performance of a Machine Learning algorithm, and the most commonly used are precision, recall, and the F1-score. \n",
    "\n",
    "<img src=\"images/pr_rec_f.png\">\n",
    "- Precision: the number of times the algorithm positively identifies a data point (known as a true positive) divided by the total number of positives (regardless of whether they were correct). High precision value means that POIs identified by the algorithm tended to be correct, while a low value means there were more false alarms (non-POIs flagged as POIs).\n",
    "- Recall: number of events you can correctly recall divided by the the number of all correct events. In other words, this means that the number of correctly identified POIs (true positives) is divided by the number of POIs in the dataset (true positives + false negatives). A false negative would be represented by labeling a POI as a non-POI. \n",
    "- F1 score: a weighted average of precision and recall. It’s calculated by multiplying the product of the recall and precision by two, and then dividing by the sum of the precision and recall. In our case, both precision and recall values of 0.3 had to be achieved using the final algorithm. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "To conclude, the following features were selected for the model: \n",
    "- f_bonus\n",
    "- deferral_payments\n",
    "- f_from_poi\n",
    "- f_salary\n",
    "- deferred_income\n",
    "- other\n",
    "- f_shared_receipt_with_poi\n",
    "- from_messages\n",
    "- expenses\n",
    "- from_this_person_to_poi\n",
    "- long_term_incentive\n",
    "- total_payments\n",
    "- f_exercised_stock_options\n",
    "- f_restricted_stock\n",
    "- total_stock_value "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decision Tree Classifier algorithm was used with the parameters min_samples_split : 2, splitter : 'best', min_samples_leaf : 1 (which is default)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Results of evaluation using cross-validation on 1000 folds:\n",
    "- Precision: 0.31834\n",
    "- Recall: 0.31150\n",
    "- F1 score: 0.31489"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}