Naive Bayes Classifier on Adult Dataset.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"0\"></a>\n",
    "# **Naive Bayes Classifier in Python**\n",
    "\n",
    "In machine learning, Naïve Bayes classification is a straightforward and powerful algorithm for the classification task. Naive Bayes Classifier to predict whether a person makes over 50K a year. \n",
    "\n",
    "This Notebook is an Curated one \n",
    "\n",
    "#### Author of Notebook: PRASHANT BANERJEE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a class=\"anchor\" id=\"0.1\"></a>\n",
    "# **Table of Contents**\n",
    "\n",
    "1.\t[Introduction to Naive Bayes algorithm](#1)\n",
    "2.\t[Naive Bayes algorithm intuition](#2)\n",
    "3.\t[Types of Naive Bayes algorithm](#3)\n",
    "4.\t[Applications of Naive Bayes algorithm](#4)\n",
    "5.\t[Import libraries](#5)\n",
    "6.\t[Import dataset](#6)\n",
    "7.\t[Exploratory data analysis](#7)\n",
    "8.\t[Declare feature vector and target variable](#8)\n",
    "9.\t[Split data into separate training and test set](#9)\n",
    "10.\t[Feature engineering](#10)\n",
    "11.\t[Feature scaling](#11)\n",
    "12.\t[Model training](#12)\n",
    "13.\t[Predict the results](#13)\n",
    "14.\t[Check accuracy score](#14)\n",
    "15.\t[Confusion matrix](#15)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **1. Introduction to Naive Bayes algorithm** <a class=\"anchor\" id=\"1\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)\n",
    "\n",
    "\n",
    "In machine learning, Naïve Bayes classification is a straightforward and powerful algorithm for the classification task. Naïve Bayes classification is based on applying Bayes’ theorem with strong independence assumption between the features.  Naïve Bayes classification produces good results when we use it for textual data analysis such as Natural Language Processing.\n",
    "\n",
    "\n",
    "Naïve Bayes models are also known as `simple Bayes` or `independent Bayes`. All these names refer to the application of Bayes’ theorem in the classifier’s decision rule. Naïve Bayes classifier applies the Bayes’ theorem in practice. This classifier brings the power of Bayes’ theorem to machine learning.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **2. Naive Bayes algorithm intuition** <a class=\"anchor\" id=\"2\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)\n",
    "\n",
    "\n",
    "Naïve Bayes Classifier uses the Bayes’ theorem to predict membership probabilities for each class such as the probability that given record or data point belongs to a particular class. The class with the highest probability is considered as the most likely class. This is also known as the **Maximum A Posteriori (MAP)**. \n",
    "\n",
    "The **MAP for a hypothesis with 2 events A and B is**\n",
    "\n",
    "**MAP (A)**\n",
    "\n",
    "= max (P (A | B))\n",
    "\n",
    "= max (P (B | A) * P (A))/P (B)\n",
    "\n",
    "= max (P (B | A) * P (A))\n",
    "\n",
    "\n",
    "Here, P (B) is evidence probability. It is used to normalize the result. It remains the same, So, removing it would not affect the result.\n",
    "\n",
    "\n",
    "Naïve Bayes Classifier assumes that all the features are unrelated to each other. Presence or absence of a feature does not influence the presence or absence of any other feature. \n",
    "\n",
    "\n",
    "In real world datasets, we test a hypothesis given multiple evidence on features. So, the calculations become quite complicated. To simplify the work, the feature independence approach is used to uncouple multiple evidence and treat each as an independent one.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **3. Types of Naive Bayes algorithm** <a class=\"anchor\" id=\"3\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)\n",
    "\n",
    "\n",
    "There are 3 types of Naïve Bayes algorithm. The 3 types are listed below:-\n",
    "\n",
    "  1. Gaussian Naïve Bayes [Explained below]\n",
    "\n",
    "  2. Multinomial Naïve Bayes\n",
    "\n",
    "  3. Bernoulli Naïve Bayes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Gaussian Naïve Bayes algorithm**\n",
    "\n",
    "\n",
    "When we have continuous attribute values, we made an assumption that the values associated with each class are distributed according to Gaussian or Normal distribution. For example, suppose the training data contains a continuous attribute x. We first segment the data by the class, and then compute the mean and variance of x in each class. Let µi be the mean of the values and let σi be the variance of the values associated with the ith class. Suppose we have some observation value xi . Then, the probability distribution of xi given a class can be computed by the following equation –\n",
    "\n",
    "\n",
    "![Gaussian Naive Bayes algorithm](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQEWCcq1XtC1Yw20KWSHn2axYa7eY-a0T1TGtdVn5PvOpv9wW3FeA&s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **4. Applications of Naive Bayes algorithm** <a class=\"anchor\" id=\"4\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)\n",
    "\n",
    "\n",
    "\n",
    "Naïve Bayes is one of the most straightforward and fast classification algorithm. It is very well suited for large volume of data. It is successfully used in various applications such as :\n",
    "\n",
    "1. Spam filtering\n",
    "2. Text classification\n",
    "3. Sentiment analysis\n",
    "4. Recommender systems\n",
    "\n",
    "It uses Bayes theorem of probability for prediction of unknown class.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **5. Import libraries** <a class=\"anchor\" id=\"5\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This Python 3 environment comes with many helpful analytics libraries installed\n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "import matplotlib.pyplot as plt # for data visualization purposes\n",
    "import seaborn as sns # for statistical data visualization\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **6. Import dataset** <a class=\"anchor\" id=\"6\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0                 1       2          3   4                   5   \\\n",
       "0  39         State-gov   77516  Bachelors  13       Never-married   \n",
       "1  50  Self-emp-not-inc   83311  Bachelors  13  Married-civ-spouse   \n",
       "2  38           Private  215646    HS-grad   9            Divorced   \n",
       "3  53           Private  234721       11th   7  Married-civ-spouse   \n",
       "4  28           Private  338409  Bachelors  13  Married-civ-spouse   \n",
       "\n",
       "                  6              7      8       9     10  11  12  \\\n",
       "0       Adm-clerical  Not-in-family  White    Male  2174   0  40   \n",
       "1    Exec-managerial        Husband  White    Male     0   0  13   \n",
       "2  Handlers-cleaners  Not-in-family  White    Male     0   0  40   \n",
       "3  Handlers-cleaners        Husband  Black    Male     0   0  40   \n",
       "4     Prof-specialty           Wife  Black  Female     0   0  40   \n",
       "\n",
       "              13     14  \n",
       "0  United-States  <=50K  \n",
       "1  United-States  <=50K  \n",
       "2  United-States  <=50K  \n",
       "3  United-States  <=50K  \n",
       "4           Cuba  <=50K  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = '/Users/ashwinimathur/Desktop/Naive Bayes/adult.csv'\n",
    "\n",
    "df = pd.read_csv(data, header=None, sep=',\\s')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **7. Exploratory data analysis** <a class=\"anchor\" id=\"7\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)\n",
    "\n",
    "\n",
    "Now, I will explore the data to gain insights about the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(32561, 15)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view dimensions of dataset\n",
    "\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are 32561 instances and 15 attributes in the data set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View top 5 rows of dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0                 1       2          3   4                   5   \\\n",
       "0  39         State-gov   77516  Bachelors  13       Never-married   \n",
       "1  50  Self-emp-not-inc   83311  Bachelors  13  Married-civ-spouse   \n",
       "2  38           Private  215646    HS-grad   9            Divorced   \n",
       "3  53           Private  234721       11th   7  Married-civ-spouse   \n",
       "4  28           Private  338409  Bachelors  13  Married-civ-spouse   \n",
       "\n",
       "                  6              7      8       9     10  11  12  \\\n",
       "0       Adm-clerical  Not-in-family  White    Male  2174   0  40   \n",
       "1    Exec-managerial        Husband  White    Male     0   0  13   \n",
       "2  Handlers-cleaners  Not-in-family  White    Male     0   0  40   \n",
       "3  Handlers-cleaners        Husband  Black    Male     0   0  40   \n",
       "4     Prof-specialty           Wife  Black  Female     0   0  40   \n",
       "\n",
       "              13     14  \n",
       "0  United-States  <=50K  \n",
       "1  United-States  <=50K  \n",
       "2  United-States  <=50K  \n",
       "3  United-States  <=50K  \n",
       "4           Cuba  <=50K  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# preview the dataset\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rename column names\n",
    "\n",
    "We can see that the dataset does not have proper column names. The columns are merely labelled as 0,1,2.... and so on. We should give proper names to the columns. I will do it as follows:-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['age', 'workclass', 'fnlwgt', 'education', 'education_num',\n",
       "       'marital_status', 'occupation', 'relationship', 'race', 'sex',\n",
       "       'capital_gain', 'capital_loss', 'hours_per_week', 'native_country',\n",
       "       'income'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_names = ['age', 'workclass', 'fnlwgt', 'education', 'education_num', 'marital_status', 'occupation', 'relationship',\n",
    "             'race', 'sex', 'capital_gain', 'capital_loss', 'hours_per_week', 'native_country', 'income']\n",
    "\n",
    "df.columns = col_names\n",
    "\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>education_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital_gain</th>\n",
       "      <th>capital_loss</th>\n",
       "      <th>hours_per_week</th>\n",
       "      <th>native_country</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>State-gov</td>\n",
       "      <td>77516</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>83311</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>215646</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>Private</td>\n",
       "      <td>234721</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>338409</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age         workclass  fnlwgt  education  education_num  \\\n",
       "0   39         State-gov   77516  Bachelors             13   \n",
       "1   50  Self-emp-not-inc   83311  Bachelors             13   \n",
       "2   38           Private  215646    HS-grad              9   \n",
       "3   53           Private  234721       11th              7   \n",
       "4   28           Private  338409  Bachelors             13   \n",
       "\n",
       "       marital_status         occupation   relationship   race     sex  \\\n",
       "0       Never-married       Adm-clerical  Not-in-family  White    Male   \n",
       "1  Married-civ-spouse    Exec-managerial        Husband  White    Male   \n",
       "2            Divorced  Handlers-cleaners  Not-in-family  White    Male   \n",
       "3  Married-civ-spouse  Handlers-cleaners        Husband  Black    Male   \n",
       "4  Married-civ-spouse     Prof-specialty           Wife  Black  Female   \n",
       "\n",
       "   capital_gain  capital_loss  hours_per_week native_country income  \n",
       "0          2174             0              40  United-States  <=50K  \n",
       "1             0             0              13  United-States  <=50K  \n",
       "2             0             0              40  United-States  <=50K  \n",
       "3             0             0              40  United-States  <=50K  \n",
       "4             0             0              40           Cuba  <=50K  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's again preview the dataset\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the column names are renamed. Now, the columns have meaningful names."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View summary of dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 32561 entries, 0 to 32560\n",
      "Data columns (total 15 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   age             32561 non-null  int64 \n",
      " 1   workclass       32561 non-null  object\n",
      " 2   fnlwgt          32561 non-null  int64 \n",
      " 3   education       32561 non-null  object\n",
      " 4   education_num   32561 non-null  int64 \n",
      " 5   marital_status  32561 non-null  object\n",
      " 6   occupation      32561 non-null  object\n",
      " 7   relationship    32561 non-null  object\n",
      " 8   race            32561 non-null  object\n",
      " 9   sex             32561 non-null  object\n",
      " 10  capital_gain    32561 non-null  int64 \n",
      " 11  capital_loss    32561 non-null  int64 \n",
      " 12  hours_per_week  32561 non-null  int64 \n",
      " 13  native_country  32561 non-null  object\n",
      " 14  income          32561 non-null  object\n",
      "dtypes: int64(6), object(9)\n",
      "memory usage: 3.7+ MB\n"
     ]
    }
   ],
   "source": [
    "# view summary of dataset\n",
    "\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are no missing values in the dataset. I will confirm this further."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Types of variables\n",
    "\n",
    "\n",
    "There is a given segregated dataset into categorical and numerical variables. There are a mixture of categorical and numerical variables in the dataset. Categorical variables have data type object. Numerical variables have data type int64.\n",
    "\n",
    "\n",
    "First of all, I will explore categorical variables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 9 categorical variables\n",
      "\n",
      "The categorical variables are :\n",
      "\n",
      " ['workclass', 'education', 'marital_status', 'occupation', 'relationship', 'race', 'sex', 'native_country', 'income']\n"
     ]
    }
   ],
   "source": [
    "# find categorical variables\n",
    "\n",
    "categorical = [var for var in df.columns if df[var].dtype=='O'] #Object Data Type\n",
    "\n",
    "print('There are {} categorical variables\\n'.format(len(categorical)))\n",
    "\n",
    "print('The categorical variables are :\\n\\n', categorical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>workclass</th>\n",
       "      <th>education</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>native_country</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>State-gov</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Self-emp-not-inc</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Exec-managerial</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Private</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Private</td>\n",
       "      <td>11th</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Handlers-cleaners</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Private</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>Cuba</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          workclass  education      marital_status         occupation  \\\n",
       "0         State-gov  Bachelors       Never-married       Adm-clerical   \n",
       "1  Self-emp-not-inc  Bachelors  Married-civ-spouse    Exec-managerial   \n",
       "2           Private    HS-grad            Divorced  Handlers-cleaners   \n",
       "3           Private       11th  Married-civ-spouse  Handlers-cleaners   \n",
       "4           Private  Bachelors  Married-civ-spouse     Prof-specialty   \n",
       "\n",
       "    relationship   race     sex native_country income  \n",
       "0  Not-in-family  White    Male  United-States  <=50K  \n",
       "1        Husband  White    Male  United-States  <=50K  \n",
       "2  Not-in-family  White    Male  United-States  <=50K  \n",
       "3        Husband  Black    Male  United-States  <=50K  \n",
       "4           Wife  Black  Female           Cuba  <=50K  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the categorical variables\n",
    "\n",
    "df[categorical].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of categorical variables\n",
    "\n",
    "\n",
    "- There are 9 categorical variables. \n",
    "\n",
    "\n",
    "- The categorical variables are given by `workclass`, `education`, `marital_status`, `occupation`, `relationship`, `race`, `sex`, `native_country` and `income`.\n",
    "\n",
    "\n",
    "- `income` is the target variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore problems within categorical variables\n",
    "\n",
    "\n",
    "First, I will explore the categorical variables.\n",
    "\n",
    "\n",
    "### Missing values in categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "workclass         0\n",
       "education         0\n",
       "marital_status    0\n",
       "occupation        0\n",
       "relationship      0\n",
       "race              0\n",
       "sex               0\n",
       "native_country    0\n",
       "income            0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check missing values in categorical variables\n",
    "\n",
    "df[categorical].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are no missing values in the categorical variables. I will confirm this further."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency counts of categorical variables\n",
    "\n",
    "\n",
    "Now, I will check the frequency counts of categorical variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Private             22696\n",
      "Self-emp-not-inc     2541\n",
      "Local-gov            2093\n",
      "?                    1836\n",
      "State-gov            1298\n",
      "Self-emp-inc         1116\n",
      "Federal-gov           960\n",
      "Without-pay            14\n",
      "Never-worked            7\n",
      "Name: workclass, dtype: int64\n",
      "HS-grad         10501\n",
      "Some-college     7291\n",
      "Bachelors        5355\n",
      "Masters          1723\n",
      "Assoc-voc        1382\n",
      "11th             1175\n",
      "Assoc-acdm       1067\n",
      "10th              933\n",
      "7th-8th           646\n",
      "Prof-school       576\n",
      "9th               514\n",
      "12th              433\n",
      "Doctorate         413\n",
      "5th-6th           333\n",
      "1st-4th           168\n",
      "Preschool          51\n",
      "Name: education, dtype: int64\n",
      "Married-civ-spouse       14976\n",
      "Never-married            10683\n",
      "Divorced                  4443\n",
      "Separated                 1025\n",
      "Widowed                    993\n",
      "Married-spouse-absent      418\n",
      "Married-AF-spouse           23\n",
      "Name: marital_status, dtype: int64\n",
      "Prof-specialty       4140\n",
      "Craft-repair         4099\n",
      "Exec-managerial      4066\n",
      "Adm-clerical         3770\n",
      "Sales                3650\n",
      "Other-service        3295\n",
      "Machine-op-inspct    2002\n",
      "?                    1843\n",
      "Transport-moving     1597\n",
      "Handlers-cleaners    1370\n",
      "Farming-fishing       994\n",
      "Tech-support          928\n",
      "Protective-serv       649\n",
      "Priv-house-serv       149\n",
      "Armed-Forces            9\n",
      "Name: occupation, dtype: int64\n",
      "Husband           13193\n",
      "Not-in-family      8305\n",
      "Own-child          5068\n",
      "Unmarried          3446\n",
      "Wife               1568\n",
      "Other-relative      981\n",
      "Name: relationship, dtype: int64\n",
      "White                 27816\n",
      "Black                  3124\n",
      "Asian-Pac-Islander     1039\n",
      "Amer-Indian-Eskimo      311\n",
      "Other                   271\n",
      "Name: race, dtype: int64\n",
      "Male      21790\n",
      "Female    10771\n",
      "Name: sex, dtype: int64\n",
      "United-States                 29170\n",
      "Mexico                          643\n",
      "?                               583\n",
      "Philippines                     198\n",
      "Germany                         137\n",
      "Canada                          121\n",
      "Puerto-Rico                     114\n",
      "El-Salvador                     106\n",
      "India                           100\n",
      "Cuba                             95\n",
      "England                          90\n",
      "Jamaica                          81\n",
      "South                            80\n",
      "China                            75\n",
      "Italy                            73\n",
      "Dominican-Republic               70\n",
      "Vietnam                          67\n",
      "Guatemala                        64\n",
      "Japan                            62\n",
      "Poland                           60\n",
      "Columbia                         59\n",
      "Taiwan                           51\n",
      "Haiti                            44\n",
      "Iran                             43\n",
      "Portugal                         37\n",
      "Nicaragua                        34\n",
      "Peru                             31\n",
      "France                           29\n",
      "Greece                           29\n",
      "Ecuador                          28\n",
      "Ireland                          24\n",
      "Hong                             20\n",
      "Cambodia                         19\n",
      "Trinadad&Tobago                  19\n",
      "Laos                             18\n",
      "Thailand                         18\n",
      "Yugoslavia                       16\n",
      "Outlying-US(Guam-USVI-etc)       14\n",
      "Honduras                         13\n",
      "Hungary                          13\n",
      "Scotland                         12\n",
      "Holand-Netherlands                1\n",
      "Name: native_country, dtype: int64\n",
      "<=50K    24720\n",
      ">50K      7841\n",
      "Name: income, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# view frequency counts of values in categorical variables\n",
    "\n",
    "for var in categorical: \n",
    "    \n",
    "    print(df[var].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Private             0.697030\n",
      "Self-emp-not-inc    0.078038\n",
      "Local-gov           0.064279\n",
      "?                   0.056386\n",
      "State-gov           0.039864\n",
      "Self-emp-inc        0.034274\n",
      "Federal-gov         0.029483\n",
      "Without-pay         0.000430\n",
      "Never-worked        0.000215\n",
      "Name: workclass, dtype: float64\n",
      "HS-grad         0.322502\n",
      "Some-college    0.223918\n",
      "Bachelors       0.164461\n",
      "Masters         0.052916\n",
      "Assoc-voc       0.042443\n",
      "11th            0.036086\n",
      "Assoc-acdm      0.032769\n",
      "10th            0.028654\n",
      "7th-8th         0.019840\n",
      "Prof-school     0.017690\n",
      "9th             0.015786\n",
      "12th            0.013298\n",
      "Doctorate       0.012684\n",
      "5th-6th         0.010227\n",
      "1st-4th         0.005160\n",
      "Preschool       0.001566\n",
      "Name: education, dtype: float64\n",
      "Married-civ-spouse       0.459937\n",
      "Never-married            0.328092\n",
      "Divorced                 0.136452\n",
      "Separated                0.031479\n",
      "Widowed                  0.030497\n",
      "Married-spouse-absent    0.012837\n",
      "Married-AF-spouse        0.000706\n",
      "Name: marital_status, dtype: float64\n",
      "Prof-specialty       0.127146\n",
      "Craft-repair         0.125887\n",
      "Exec-managerial      0.124873\n",
      "Adm-clerical         0.115783\n",
      "Sales                0.112097\n",
      "Other-service        0.101195\n",
      "Machine-op-inspct    0.061485\n",
      "?                    0.056601\n",
      "Transport-moving     0.049046\n",
      "Handlers-cleaners    0.042075\n",
      "Farming-fishing      0.030527\n",
      "Tech-support         0.028500\n",
      "Protective-serv      0.019932\n",
      "Priv-house-serv      0.004576\n",
      "Armed-Forces         0.000276\n",
      "Name: occupation, dtype: float64\n",
      "Husband           0.405178\n",
      "Not-in-family     0.255060\n",
      "Own-child         0.155646\n",
      "Unmarried         0.105832\n",
      "Wife              0.048156\n",
      "Other-relative    0.030128\n",
      "Name: relationship, dtype: float64\n",
      "White                 0.854274\n",
      "Black                 0.095943\n",
      "Asian-Pac-Islander    0.031909\n",
      "Amer-Indian-Eskimo    0.009551\n",
      "Other                 0.008323\n",
      "Name: race, dtype: float64\n",
      "Male      0.669205\n",
      "Female    0.330795\n",
      "Name: sex, dtype: float64\n",
      "United-States                 0.895857\n",
      "Mexico                        0.019748\n",
      "?                             0.017905\n",
      "Philippines                   0.006081\n",
      "Germany                       0.004207\n",
      "Canada                        0.003716\n",
      "Puerto-Rico                   0.003501\n",
      "El-Salvador                   0.003255\n",
      "India                         0.003071\n",
      "Cuba                          0.002918\n",
      "England                       0.002764\n",
      "Jamaica                       0.002488\n",
      "South                         0.002457\n",
      "China                         0.002303\n",
      "Italy                         0.002242\n",
      "Dominican-Republic            0.002150\n",
      "Vietnam                       0.002058\n",
      "Guatemala                     0.001966\n",
      "Japan                         0.001904\n",
      "Poland                        0.001843\n",
      "Columbia                      0.001812\n",
      "Taiwan                        0.001566\n",
      "Haiti                         0.001351\n",
      "Iran                          0.001321\n",
      "Portugal                      0.001136\n",
      "Nicaragua                     0.001044\n",
      "Peru                          0.000952\n",
      "France                        0.000891\n",
      "Greece                        0.000891\n",
      "Ecuador                       0.000860\n",
      "Ireland                       0.000737\n",
      "Hong                          0.000614\n",
      "Cambodia                      0.000584\n",
      "Trinadad&Tobago               0.000584\n",
      "Laos                          0.000553\n",
      "Thailand                      0.000553\n",
      "Yugoslavia                    0.000491\n",
      "Outlying-US(Guam-USVI-etc)    0.000430\n",
      "Honduras                      0.000399\n",
      "Hungary                       0.000399\n",
      "Scotland                      0.000369\n",
      "Holand-Netherlands            0.000031\n",
      "Name: native_country, dtype: float64\n",
      "<=50K    0.75919\n",
      ">50K     0.24081\n",
      "Name: income, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "# view frequency distribution of categorical variables\n",
    "\n",
    "for var in categorical: \n",
    "    \n",
    "    print(df[var].value_counts()/np.float(len(df)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can see that there are several variables like `workclass`, `occupation` and `native_country` which contain missing values. Generally, the missing values are coded as `NaN` and python will detect them with the usual command of `df.isnull().sum()`.\n",
    "\n",
    "But, in this case the missing values are coded as `?`. Python fail to detect these as missing values because it do not consider `?` as missing values. So, I have to replace `?` with `NaN` so that Python can detect these missing values.\n",
    "\n",
    "I will explore these variables and replace `?` with `NaN`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore workclass variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['State-gov', 'Self-emp-not-inc', 'Private', 'Federal-gov',\n",
       "       'Local-gov', '?', 'Self-emp-inc', 'Without-pay', 'Never-worked'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check labels in workclass variable\n",
    "\n",
    "df.workclass.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Private             22696\n",
       "Self-emp-not-inc     2541\n",
       "Local-gov            2093\n",
       "?                    1836\n",
       "State-gov            1298\n",
       "Self-emp-inc         1116\n",
       "Federal-gov           960\n",
       "Without-pay            14\n",
       "Never-worked            7\n",
       "Name: workclass, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check frequency distribution of values in workclass variable\n",
    "\n",
    "df.workclass.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are 1836 values encoded as `?` in workclass variable. I will replace these `?` with `NaN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace '?' values in workclass variable with `NaN`\n",
    "\n",
    "\n",
    "df['workclass'].replace('?', np.NaN, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Private             22696\n",
       "Self-emp-not-inc     2541\n",
       "Local-gov            2093\n",
       "State-gov            1298\n",
       "Self-emp-inc         1116\n",
       "Federal-gov           960\n",
       "Without-pay            14\n",
       "Never-worked            7\n",
       "Name: workclass, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# again check the frequency distribution of values in workclass variable\n",
    "\n",
    "df.workclass.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can see that there are no values encoded as `?` in the `workclass` variable.\n",
    "\n",
    "I will adopt similar approach with `occupation` and `native_country` column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore occupation variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Adm-clerical', 'Exec-managerial', 'Handlers-cleaners',\n",
       "       'Prof-specialty', 'Other-service', 'Sales', 'Craft-repair',\n",
       "       'Transport-moving', 'Farming-fishing', 'Machine-op-inspct',\n",
       "       'Tech-support', '?', 'Protective-serv', 'Armed-Forces',\n",
       "       'Priv-house-serv'], dtype=object)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check labels in occupation variable\n",
    "\n",
    "df.occupation.unique()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prof-specialty       4140\n",
       "Craft-repair         4099\n",
       "Exec-managerial      4066\n",
       "Adm-clerical         3770\n",
       "Sales                3650\n",
       "Other-service        3295\n",
       "Machine-op-inspct    2002\n",
       "?                    1843\n",
       "Transport-moving     1597\n",
       "Handlers-cleaners    1370\n",
       "Farming-fishing       994\n",
       "Tech-support          928\n",
       "Protective-serv       649\n",
       "Priv-house-serv       149\n",
       "Armed-Forces            9\n",
       "Name: occupation, dtype: int64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check frequency distribution of values in occupation variable\n",
    "\n",
    "df.occupation.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are 1843 values encoded as `?` in `occupation` variable. I will replace these `?` with `NaN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace '?' values in occupation variable with `NaN`\n",
    "\n",
    "df['occupation'].replace('?', np.NaN, inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prof-specialty       4140\n",
       "Craft-repair         4099\n",
       "Exec-managerial      4066\n",
       "Adm-clerical         3770\n",
       "Sales                3650\n",
       "Other-service        3295\n",
       "Machine-op-inspct    2002\n",
       "Transport-moving     1597\n",
       "Handlers-cleaners    1370\n",
       "Farming-fishing       994\n",
       "Tech-support          928\n",
       "Protective-serv       649\n",
       "Priv-house-serv       149\n",
       "Armed-Forces            9\n",
       "Name: occupation, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# again check the frequency distribution of values in occupation variable\n",
    "\n",
    "df.occupation.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore native_country variable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['United-States', 'Cuba', 'Jamaica', 'India', '?', 'Mexico',\n",
       "       'South', 'Puerto-Rico', 'Honduras', 'England', 'Canada', 'Germany',\n",
       "       'Iran', 'Philippines', 'Italy', 'Poland', 'Columbia', 'Cambodia',\n",
       "       'Thailand', 'Ecuador', 'Laos', 'Taiwan', 'Haiti', 'Portugal',\n",
       "       'Dominican-Republic', 'El-Salvador', 'France', 'Guatemala',\n",
       "       'China', 'Japan', 'Yugoslavia', 'Peru',\n",
       "       'Outlying-US(Guam-USVI-etc)', 'Scotland', 'Trinadad&Tobago',\n",
       "       'Greece', 'Nicaragua', 'Vietnam', 'Hong', 'Ireland', 'Hungary',\n",
       "       'Holand-Netherlands'], dtype=object)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check labels in native_country variable\n",
    "\n",
    "df.native_country.unique()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "United-States                 29170\n",
       "Mexico                          643\n",
       "?                               583\n",
       "Philippines                     198\n",
       "Germany                         137\n",
       "Canada                          121\n",
       "Puerto-Rico                     114\n",
       "El-Salvador                     106\n",
       "India                           100\n",
       "Cuba                             95\n",
       "England                          90\n",
       "Jamaica                          81\n",
       "South                            80\n",
       "China                            75\n",
       "Italy                            73\n",
       "Dominican-Republic               70\n",
       "Vietnam                          67\n",
       "Guatemala                        64\n",
       "Japan                            62\n",
       "Poland                           60\n",
       "Columbia                         59\n",
       "Taiwan                           51\n",
       "Haiti                            44\n",
       "Iran                             43\n",
       "Portugal                         37\n",
       "Nicaragua                        34\n",
       "Peru                             31\n",
       "France                           29\n",
       "Greece                           29\n",
       "Ecuador                          28\n",
       "Ireland                          24\n",
       "Hong                             20\n",
       "Cambodia                         19\n",
       "Trinadad&Tobago                  19\n",
       "Laos                             18\n",
       "Thailand                         18\n",
       "Yugoslavia                       16\n",
       "Outlying-US(Guam-USVI-etc)       14\n",
       "Honduras                         13\n",
       "Hungary                          13\n",
       "Scotland                         12\n",
       "Holand-Netherlands                1\n",
       "Name: native_country, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check frequency distribution of values in native_country variable\n",
    "\n",
    "df.native_country.value_counts()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are 583 values encoded as `?` in `native_country` variable. I will replace these `?` with `NaN`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace '?' values in native_country variable with `NaN`\n",
    "\n",
    "df['native_country'].replace('?', np.NaN, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "United-States                 29170\n",
       "Mexico                          643\n",
       "Philippines                     198\n",
       "Germany                         137\n",
       "Canada                          121\n",
       "Puerto-Rico                     114\n",
       "El-Salvador                     106\n",
       "India                           100\n",
       "Cuba                             95\n",
       "England                          90\n",
       "Jamaica                          81\n",
       "South                            80\n",
       "China                            75\n",
       "Italy                            73\n",
       "Dominican-Republic               70\n",
       "Vietnam                          67\n",
       "Guatemala                        64\n",
       "Japan                            62\n",
       "Poland                           60\n",
       "Columbia                         59\n",
       "Taiwan                           51\n",
       "Haiti                            44\n",
       "Iran                             43\n",
       "Portugal                         37\n",
       "Nicaragua                        34\n",
       "Peru                             31\n",
       "France                           29\n",
       "Greece                           29\n",
       "Ecuador                          28\n",
       "Ireland                          24\n",
       "Hong                             20\n",
       "Cambodia                         19\n",
       "Trinadad&Tobago                  19\n",
       "Laos                             18\n",
       "Thailand                         18\n",
       "Yugoslavia                       16\n",
       "Outlying-US(Guam-USVI-etc)       14\n",
       "Honduras                         13\n",
       "Hungary                          13\n",
       "Scotland                         12\n",
       "Holand-Netherlands                1\n",
       "Name: native_country, dtype: int64"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# again check the frequency distribution of values in native_country variable\n",
    "\n",
    "df.native_country.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check missing values in categorical variables again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "workclass         1836\n",
       "education            0\n",
       "marital_status       0\n",
       "occupation        1843\n",
       "relationship         0\n",
       "race                 0\n",
       "sex                  0\n",
       "native_country     583\n",
       "income               0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[categorical].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can see that `workclass`, `occupation` and `native_country` variable contains missing values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of labels: cardinality\n",
    "\n",
    "\n",
    "The number of labels within a categorical variable is known as **cardinality**. A high number of labels within a variable is known as **high cardinality**. High cardinality may pose some serious problems in the machine learning model. So, I will check for high cardinality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "workclass  contains  9  labels\n",
      "education  contains  16  labels\n",
      "marital_status  contains  7  labels\n",
      "occupation  contains  15  labels\n",
      "relationship  contains  6  labels\n",
      "race  contains  5  labels\n",
      "sex  contains  2  labels\n",
      "native_country  contains  42  labels\n",
      "income  contains  2  labels\n"
     ]
    }
   ],
   "source": [
    "# check for cardinality in categorical variables\n",
    "\n",
    "for var in categorical:\n",
    "    \n",
    "    print(var, ' contains ', len(df[var].unique()), ' labels')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that `native_country` column contains relatively large number of labels as compared to other columns. I will check for cardinality after train-test split."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore Numerical Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 6 numerical variables\n",
      "\n",
      "The numerical variables are : ['age', 'fnlwgt', 'education_num', 'capital_gain', 'capital_loss', 'hours_per_week']\n"
     ]
    }
   ],
   "source": [
    "# find numerical variables\n",
    "\n",
    "numerical = [var for var in df.columns if df[var].dtype!='O']\n",
    "\n",
    "print('There are {} numerical variables\\n'.format(len(numerical)))\n",
    "\n",
    "print('The numerical variables are :', numerical)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education_num</th>\n",
       "      <th>capital_gain</th>\n",
       "      <th>capital_loss</th>\n",
       "      <th>hours_per_week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39</td>\n",
       "      <td>77516</td>\n",
       "      <td>13</td>\n",
       "      <td>2174</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>83311</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>215646</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>53</td>\n",
       "      <td>234721</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>338409</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  fnlwgt  education_num  capital_gain  capital_loss  hours_per_week\n",
       "0   39   77516             13          2174             0              40\n",
       "1   50   83311             13             0             0              13\n",
       "2   38  215646              9             0             0              40\n",
       "3   53  234721              7             0             0              40\n",
       "4   28  338409             13             0             0              40"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the numerical variables\n",
    "\n",
    "df[numerical].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of numerical variables\n",
    "\n",
    "\n",
    "- There are 6 numerical variables. \n",
    "\n",
    "\n",
    "- These are given by `age`, `fnlwgt`, `education_num`, `capital_gain`, `capital_loss` and `hours_per_week`.\n",
    "\n",
    "\n",
    "- All of the numerical variables are of discrete data type."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore problems within numerical variables\n",
    "\n",
    "\n",
    "Now, I will explore the numerical variables.\n",
    "\n",
    "\n",
    "### Missing values in numerical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age               0\n",
       "fnlwgt            0\n",
       "education_num     0\n",
       "capital_gain      0\n",
       "capital_loss      0\n",
       "hours_per_week    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check missing values in numerical variables\n",
    "\n",
    "df[numerical].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that all the 6 numerical variables do not contain missing values. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **8. Declare feature vector and target variable** <a class=\"anchor\" id=\"8\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(['income'], axis=1)\n",
    "\n",
    "y = df['income']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **9. Split data into separate training and test set** <a class=\"anchor\" id=\"9\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split X and y into training and testing sets\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((22792, 14), (9769, 14))"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check the shape of X_train and X_test\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **10. Feature Engineering** <a class=\"anchor\" id=\"10\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)\n",
    "\n",
    "\n",
    "**Feature Engineering** is the process of transforming raw data into useful features that help us to understand our model better and increase its predictive power. I will carry out feature engineering on different types of variables.\n",
    "\n",
    "\n",
    "First, I will display the categorical and numerical variables again separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age                int64\n",
       "workclass         object\n",
       "fnlwgt             int64\n",
       "education         object\n",
       "education_num      int64\n",
       "marital_status    object\n",
       "occupation        object\n",
       "relationship      object\n",
       "race              object\n",
       "sex               object\n",
       "capital_gain       int64\n",
       "capital_loss       int64\n",
       "hours_per_week     int64\n",
       "native_country    object\n",
       "dtype: object"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check data types in X_train\n",
    "\n",
    "X_train.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['workclass',\n",
       " 'education',\n",
       " 'marital_status',\n",
       " 'occupation',\n",
       " 'relationship',\n",
       " 'race',\n",
       " 'sex',\n",
       " 'native_country']"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# display categorical variables\n",
    "\n",
    "categorical = [col for col in X_train.columns if X_train[col].dtypes == 'O']\n",
    "\n",
    "categorical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['age',\n",
       " 'fnlwgt',\n",
       " 'education_num',\n",
       " 'capital_gain',\n",
       " 'capital_loss',\n",
       " 'hours_per_week']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# display numerical variables\n",
    "\n",
    "numerical = [col for col in X_train.columns if X_train[col].dtypes != 'O']\n",
    "\n",
    "numerical"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Engineering missing values in categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "workclass         0.055985\n",
       "education         0.000000\n",
       "marital_status    0.000000\n",
       "occupation        0.056072\n",
       "relationship      0.000000\n",
       "race              0.000000\n",
       "sex               0.000000\n",
       "native_country    0.018164\n",
       "dtype: float64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# print percentage of missing values in the categorical variables in training set\n",
    "\n",
    "X_train[categorical].isnull().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "workclass 0.055984555984555984\n",
      "occupation 0.05607230607230607\n",
      "native_country 0.018164268164268166\n"
     ]
    }
   ],
   "source": [
    "# print categorical variables with missing data\n",
    "\n",
    "for col in categorical:\n",
    "    if X_train[col].isnull().mean()>0:\n",
    "        print(col, (X_train[col].isnull().mean()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# impute missing categorical variables with most frequent value\n",
    "\n",
    "for df2 in [X_train, X_test]:\n",
    "    df2['workclass'].fillna(X_train['workclass'].mode()[0], inplace=True)\n",
    "    df2['occupation'].fillna(X_train['occupation'].mode()[0], inplace=True)\n",
    "    df2['native_country'].fillna(X_train['native_country'].mode()[0], inplace=True)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "workclass         0\n",
       "education         0\n",
       "marital_status    0\n",
       "occupation        0\n",
       "relationship      0\n",
       "race              0\n",
       "sex               0\n",
       "native_country    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check missing values in categorical variables in X_train\n",
    "\n",
    "X_train[categorical].isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "workclass         0\n",
       "education         0\n",
       "marital_status    0\n",
       "occupation        0\n",
       "relationship      0\n",
       "race              0\n",
       "sex               0\n",
       "native_country    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check missing values in categorical variables in X_test\n",
    "\n",
    "X_test[categorical].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a final check, I will check for missing values in X_train and X_test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age               0\n",
       "workclass         0\n",
       "fnlwgt            0\n",
       "education         0\n",
       "education_num     0\n",
       "marital_status    0\n",
       "occupation        0\n",
       "relationship      0\n",
       "race              0\n",
       "sex               0\n",
       "capital_gain      0\n",
       "capital_loss      0\n",
       "hours_per_week    0\n",
       "native_country    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check missing values in X_train\n",
    "\n",
    "X_train.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "age               0\n",
       "workclass         0\n",
       "fnlwgt            0\n",
       "education         0\n",
       "education_num     0\n",
       "marital_status    0\n",
       "occupation        0\n",
       "relationship      0\n",
       "race              0\n",
       "sex               0\n",
       "capital_gain      0\n",
       "capital_loss      0\n",
       "hours_per_week    0\n",
       "native_country    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check missing values in X_test\n",
    "\n",
    "X_test.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there are no missing values in X_train and X_test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/scikit-learn-contrib/category_encoders\n",
      "  Cloning https://github.com/scikit-learn-contrib/category_encoders to /private/var/folders/d6/nj2wf6wd1x9bbpjwb2xf3vcw0000gp/T/pip-req-build-uqojg3hm\n",
      "  Running command git clone -q https://github.com/scikit-learn-contrib/category_encoders /private/var/folders/d6/nj2wf6wd1x9bbpjwb2xf3vcw0000gp/T/pip-req-build-uqojg3hm\n",
      "  Resolved https://github.com/scikit-learn-contrib/category_encoders to commit 8f7628d7b0ce87128e53a53e89d3c0156b73d46e\n",
      "Requirement already satisfied: numpy>=1.14.0 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from category-encoders==2.5.1.post0) (1.21.5)\n",
      "Requirement already satisfied: scikit-learn>=0.20.0 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from category-encoders==2.5.1.post0) (1.0.2)\n",
      "Requirement already satisfied: scipy>=1.0.0 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from category-encoders==2.5.1.post0) (1.7.3)\n",
      "Requirement already satisfied: statsmodels>=0.9.0 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from category-encoders==2.5.1.post0) (0.13.2)\n",
      "Requirement already satisfied: pandas>=1.0.5 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from category-encoders==2.5.1.post0) (1.4.2)\n",
      "Requirement already satisfied: patsy>=0.5.1 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from category-encoders==2.5.1.post0) (0.5.2)\n",
      "Requirement already satisfied: python-dateutil>=2.8.1 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from pandas>=1.0.5->category-encoders==2.5.1.post0) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from pandas>=1.0.5->category-encoders==2.5.1.post0) (2021.3)\n",
      "Requirement already satisfied: six in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from patsy>=0.5.1->category-encoders==2.5.1.post0) (1.16.0)\n",
      "Requirement already satisfied: joblib>=0.11 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from scikit-learn>=0.20.0->category-encoders==2.5.1.post0) (1.1.0)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from scikit-learn>=0.20.0->category-encoders==2.5.1.post0) (2.2.0)\n",
      "Requirement already satisfied: packaging>=21.3 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from statsmodels>=0.9.0->category-encoders==2.5.1.post0) (21.3)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /Users/ashwinimathur/opt/anaconda3/lib/python3.9/site-packages (from packaging>=21.3->statsmodels>=0.9.0->category-encoders==2.5.1.post0) (3.0.4)\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade git+https://github.com/scikit-learn-contrib/category_encoders\n",
    "import category_encoders as ce"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# encode remaining variables with one-hot encoding\n",
    "\n",
    "encoder = ce.OneHotEncoder(cols=['workclass', 'education', 'marital_status', 'occupation', 'relationship', \n",
    "                                 'race', 'sex', 'native_country'])\n",
    "\n",
    "X_train = encoder.fit_transform(X_train)\n",
    "\n",
    "X_test = encoder.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass_1</th>\n",
       "      <th>workclass_2</th>\n",
       "      <th>workclass_3</th>\n",
       "      <th>workclass_4</th>\n",
       "      <th>workclass_5</th>\n",
       "      <th>workclass_6</th>\n",
       "      <th>workclass_7</th>\n",
       "      <th>workclass_8</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>...</th>\n",
       "      <th>native_country_32</th>\n",
       "      <th>native_country_33</th>\n",
       "      <th>native_country_34</th>\n",
       "      <th>native_country_35</th>\n",
       "      <th>native_country_36</th>\n",
       "      <th>native_country_37</th>\n",
       "      <th>native_country_38</th>\n",
       "      <th>native_country_39</th>\n",
       "      <th>native_country_40</th>\n",
       "      <th>native_country_41</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>32098</th>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>170871</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25206</th>\n",
       "      <td>47</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>108890</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23491</th>\n",
       "      <td>48</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>187505</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12367</th>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>145592</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7054</th>\n",
       "      <td>23</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>203003</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 105 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       age  workclass_1  workclass_2  workclass_3  workclass_4  workclass_5  \\\n",
       "32098   45            1            0            0            0            0   \n",
       "25206   47            0            1            0            0            0   \n",
       "23491   48            1            0            0            0            0   \n",
       "12367   29            1            0            0            0            0   \n",
       "7054    23            1            0            0            0            0   \n",
       "\n",
       "       workclass_6  workclass_7  workclass_8  fnlwgt  ...  native_country_32  \\\n",
       "32098            0            0            0  170871  ...                  0   \n",
       "25206            0            0            0  108890  ...                  0   \n",
       "23491            0            0            0  187505  ...                  0   \n",
       "12367            0            0            0  145592  ...                  0   \n",
       "7054             0            0            0  203003  ...                  0   \n",
       "\n",
       "       native_country_33  native_country_34  native_country_35  \\\n",
       "32098                  0                  0                  0   \n",
       "25206                  0                  0                  0   \n",
       "23491                  0                  0                  0   \n",
       "12367                  0                  0                  0   \n",
       "7054                   0                  0                  0   \n",
       "\n",
       "       native_country_36  native_country_37  native_country_38  \\\n",
       "32098                  0                  0                  0   \n",
       "25206                  0                  0                  0   \n",
       "23491                  0                  0                  0   \n",
       "12367                  0                  0                  0   \n",
       "7054                   0                  0                  0   \n",
       "\n",
       "       native_country_39  native_country_40  native_country_41  \n",
       "32098                  0                  0                  0  \n",
       "25206                  0                  0                  0  \n",
       "23491                  0                  0                  0  \n",
       "12367                  0                  0                  0  \n",
       "7054                   0                  0                  0  \n",
       "\n",
       "[5 rows x 105 columns]"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(22792, 105)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that from the initial 14 columns, we now have 113 columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, I will take a look at the `X_test` set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass_1</th>\n",
       "      <th>workclass_2</th>\n",
       "      <th>workclass_3</th>\n",
       "      <th>workclass_4</th>\n",
       "      <th>workclass_5</th>\n",
       "      <th>workclass_6</th>\n",
       "      <th>workclass_7</th>\n",
       "      <th>workclass_8</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>...</th>\n",
       "      <th>native_country_32</th>\n",
       "      <th>native_country_33</th>\n",
       "      <th>native_country_34</th>\n",
       "      <th>native_country_35</th>\n",
       "      <th>native_country_36</th>\n",
       "      <th>native_country_37</th>\n",
       "      <th>native_country_38</th>\n",
       "      <th>native_country_39</th>\n",
       "      <th>native_country_40</th>\n",
       "      <th>native_country_41</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22278</th>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>177119</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8950</th>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>216481</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7838</th>\n",
       "      <td>25</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>256263</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16505</th>\n",
       "      <td>46</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>147640</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19140</th>\n",
       "      <td>45</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>172822</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 105 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       age  workclass_1  workclass_2  workclass_3  workclass_4  workclass_5  \\\n",
       "22278   27            1            0            0            0            0   \n",
       "8950    27            1            0            0            0            0   \n",
       "7838    25            1            0            0            0            0   \n",
       "16505   46            1            0            0            0            0   \n",
       "19140   45            1            0            0            0            0   \n",
       "\n",
       "       workclass_6  workclass_7  workclass_8  fnlwgt  ...  native_country_32  \\\n",
       "22278            0            0            0  177119  ...                  0   \n",
       "8950             0            0            0  216481  ...                  0   \n",
       "7838             0            0            0  256263  ...                  0   \n",
       "16505            0            0            0  147640  ...                  0   \n",
       "19140            0            0            0  172822  ...                  0   \n",
       "\n",
       "       native_country_33  native_country_34  native_country_35  \\\n",
       "22278                  0                  0                  0   \n",
       "8950                   0                  0                  0   \n",
       "7838                   0                  0                  0   \n",
       "16505                  0                  0                  0   \n",
       "19140                  0                  0                  0   \n",
       "\n",
       "       native_country_36  native_country_37  native_country_38  \\\n",
       "22278                  0                  0                  0   \n",
       "8950                   0                  0                  0   \n",
       "7838                   0                  0                  0   \n",
       "16505                  0                  0                  0   \n",
       "19140                  0                  0                  0   \n",
       "\n",
       "       native_country_39  native_country_40  native_country_41  \n",
       "22278                  0                  0                  0  \n",
       "8950                   0                  0                  0  \n",
       "7838                   0                  0                  0  \n",
       "16505                  0                  0                  0  \n",
       "19140                  0                  0                  0  \n",
       "\n",
       "[5 rows x 105 columns]"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9769, 105)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have training and testing set ready for model building. Before that, we should map all the feature variables onto the same scale. It is called `feature scaling`. I will do it as follows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have `X_train` dataset ready to be fed into the Gaussian Naive Bayes classifier. I will do it as follows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **12. Model training** <a class=\"anchor\" id=\"12\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GaussianNB()"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train a Gaussian Naive Bayes classifier on the training set\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "\n",
    "# instantiate the model\n",
    "gnb = GaussianNB()\n",
    "\n",
    "\n",
    "# fit the model\n",
    "gnb.fit(X_train, y_train)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **13. Predict the results** <a class=\"anchor\" id=\"13\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['<=50K', '<=50K', '<=50K', ..., '>50K', '<=50K', '<=50K'],\n",
       "      dtype='<U5')"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred = gnb.predict(X_test)\n",
    "\n",
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **14. Check accuracy score** <a class=\"anchor\" id=\"14\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model accuracy score: 0.7974\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "print('Model accuracy score: {0:0.4f}'. format(accuracy_score(y_test, y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, **y_test** are the true class labels and **y_pred** are the predicted class labels in the test-set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare the train-set and test-set accuracy\n",
    "\n",
    "\n",
    "Now, I will compare the train-set and test-set accuracy to check for overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['>50K', '<=50K', '<=50K', ..., '<=50K', '>50K', '<=50K'],\n",
       "      dtype='<U5')"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_train = gnb.predict(X_train)\n",
    "\n",
    "y_pred_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training-set accuracy score: 0.7939\n"
     ]
    }
   ],
   "source": [
    "print('Training-set accuracy score: {0:0.4f}'. format(accuracy_score(y_train, y_pred_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check for overfitting and underfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set score: 0.7939\n",
      "Test set score: 0.7974\n"
     ]
    }
   ],
   "source": [
    "# print the scores on training and test set\n",
    "\n",
    "print('Training set score: {:.4f}'.format(gnb.score(X_train, y_train)))\n",
    "\n",
    "print('Test set score: {:.4f}'.format(gnb.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training-set accuracy score is 0.8067 while the test-set accuracy to be 0.8083. These two values are quite comparable. So, there is no sign of overfitting. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, based on the above analysis we can conclude that our classification model accuracy is very good. Our model is doing a very good job in terms of predicting the class labels.\n",
    "\n",
    "\n",
    "But, it does not give the underlying distribution of values. Also, it does not tell anything about the type of errors our classifer is making. \n",
    "\n",
    "\n",
    "We have another tool called `Confusion matrix` that comes to our rescue."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **15. Confusion matrix** <a class=\"anchor\" id=\"15\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)\n",
    "\n",
    "\n",
    "A confusion matrix is a tool for summarizing the performance of a classification algorithm. A confusion matrix will give us a clear picture of classification model performance and the types of errors produced by the model. It gives us a summary of correct and incorrect predictions broken down by each category. The summary is represented in a tabular form.\n",
    "\n",
    "\n",
    "Four types of outcomes are possible while evaluating a classification model performance. These four outcomes are described below:-\n",
    "\n",
    "\n",
    "**True Positives (TP)** – True Positives occur when we predict an observation belongs to a certain class and the observation actually belongs to that class.\n",
    "\n",
    "\n",
    "**True Negatives (TN)** – True Negatives occur when we predict an observation does not belong to a certain class and the observation actually does not belong to that class.\n",
    "\n",
    "\n",
    "**False Positives (FP)** – False Positives occur when we predict an observation belongs to a    certain class but the observation actually does not belong to that class. This type of error is called **Type I error.**\n",
    "\n",
    "\n",
    "\n",
    "**False Negatives (FN)** – False Negatives occur when we predict an observation does not belong to a certain class but the observation actually belongs to that class. This is a very serious error and it is called **Type II error.**\n",
    "\n",
    "\n",
    "\n",
    "These four outcomes are summarized in a confusion matrix given below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix\n",
      "\n",
      " [[7041  366]\n",
      " [1613  749]]\n",
      "\n",
      "True Positives(TP) =  7041\n",
      "\n",
      "True Negatives(TN) =  749\n",
      "\n",
      "False Positives(FP) =  366\n",
      "\n",
      "False Negatives(FN) =  1613\n"
     ]
    }
   ],
   "source": [
    "# Print the Confusion Matrix and slice it into four pieces\n",
    "\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "cm = confusion_matrix(y_test, y_pred)\n",
    "\n",
    "print('Confusion matrix\\n\\n', cm)\n",
    "\n",
    "print('\\nTrue Positives(TP) = ', cm[0,0])\n",
    "\n",
    "print('\\nTrue Negatives(TN) = ', cm[1,1])\n",
    "\n",
    "print('\\nFalse Positives(FP) = ', cm[0,1])\n",
    "\n",
    "print('\\nFalse Negatives(FN) = ', cm[1,0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The confusion matrix shows `5999 + 1897 = 7896 correct predictions` and `1408 + 465 = 1873 incorrect predictions`.\n",
    "\n",
    "\n",
    "In this case, we have\n",
    "\n",
    "\n",
    "- `True Positives` (Actual Positive:1 and Predict Positive:1) - 5999\n",
    "\n",
    "\n",
    "- `True Negatives` (Actual Negative:0 and Predict Negative:0) - 1897\n",
    "\n",
    "\n",
    "- `False Positives` (Actual Negative:0 but Predict Positive:1) - 1408 `(Type I error)`\n",
    "\n",
    "\n",
    "- `False Negatives` (Actual Positive:1 but Predict Negative:0) - 465 `(Type II error)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWcAAAD7CAYAAAC2a1UBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAoqklEQVR4nO3de5xVVf3/8dd7BlBQUbxgJF4TJSDvIUo/8i6aivrVvN++fqWUvKSlYFZWX8rSzLRQUUvMK9HXxGsZCuUVkVQENVAUUQLvoCEyw+f3x16jR5w5s2eYM3PmzPvpYz/O3mvvtffazuEza9Zeey1FBGZmVl6q2roAZmb2WQ7OZmZlyMHZzKwMOTibmZUhB2czszLk4GxmVoYcnM3M6iFpa0lPFSyLJZ0laV1J90uanT57FOQZJWmOpBck7VuQvqOkGWnf5ZLU6PXdz9nMrDhJ1cBrwM7ACODtiLhI0kigR0ScJ6kfcAswEPg88Ddgq4iolTQVOBN4DLgHuDwi7i12zU6lu51M102OcvS3z1g670dtXQQrS1s1WqNsTFNiztJ5t+S93p7AixHxiqRhwG4pfRwwGTgPGAbcGhHLgLmS5gADJb0MdI+IRwEk3QAcDLRtcDYza01SSVprjySrFQNsGBELACJigaSeKX0jsppxnfkpbXlaXzm9KLc5m1lFEVX5F2m4pGkFy/DPnE/qAhwE/LHRS39WFEkvyjVnM6soTak5R8RYYGwjh+0HTI+IhWl7oaReqdbcC1iU0ucDGxfk6w28ntJ715NelGvOZlZRpKrcS05H8UmTBsBE4IS0fgJwR0H6kZJWk7Q50AeYmppAlkgalHppHF+Qp0GuOZtZRck6VrTUudQN2Bv4RkHyRcB4SScD84DDASJipqTxwCygBhgREbUpz6nA9UBXsgeBRR8GgoOzmVWYlnwgGBH/AdZbKe0tst4b9R0/GhhdT/o0YEBTru3gbGYVpUS9NVqdg7OZVRRVyKM0B2czqyiuOZuZlSEHZzOzMlTVgr012pKDs5lVFNeczczKUKUE52bdhaRGO1CbmbWFErwh2CYarDlL2qGhXcB2JSmNmdkqK++gm1exZo0ngCnUP6LSOiUpjZnZKqqqqozW2mJ38RzwjYiYvfIOSa+WrkhmZs3XEV5CuZCG/z44veWLYma26sq9LTmvBoNzREwosu/PJSmNmdkqyjF3aruQ61fMyg8HizwsNDNrU5XSWyNv6U5tZNvMrCw0ZZqqcpbrsWZEnFJs28ysXFRKb41Gf3Uoc6ykH6TtTSQNLH3RzMyarlJqznlKNwbYhWweLYAlwG9LViIzs1WhqvxLGctT/985InaQ9E+AiHgnTRVuZlZ2yv1BX155gvNyZTMmBoCkDYAVJS2VmVkzVUpXujzB+XLgdqCnpNHAYcAFJS2VmVkzlXtbcl6NBueIuEnSk2SzzQo4OCKeK3nJzMyaQVUdZLB9Sb8GbosIPwQ0s/JXGRXnXLcxHbhA0hxJF0vaqdSFMjNrNin/UsYaDc4RMS4i9gcGAv8Cfi7pMyPVmZmVhRYMzpLWkTRB0vOSnpO0i6R1Jd0vaXb67FFw/KhUkX1B0r4F6TtKmpH2Xa4cTy2b8gfAlkBfYDPg+SbkMzNrPVVNWBr3a+C+iOgLbEs2lPJIYFJE9AEmpW0k9QOOBPoDQ4ExqacbwJXAcKBPWobmuY2iJNXVlH8MzAR2jIgDc92WmVkriyrlXoqR1B0YAlwHEBEfRcS7wDBgXDpsHHBwWh8G3BoRyyJiLjAHGCipF9A9Ih6NiABuKMjToDxd6eYCu0TEmzmONTNrW40E3UKShpPVaOuMjYixaX0L4A3g95K2BZ4EzgQ2jIgFABGxQFLPdPxGwGMF55qf0pan9ZXTiyo2h2DfiHgemApsImmTwv0RMb2xk5uZtbomPOhLgXhsA7s7ATsAp0fE46nn2shiV67vEkXSiypWcz6b7DfKLxs48R6NndzMrNW1XCeM+cD8iHg8bU8gC84LJfVKteZewKKC4zcuyN8beD2l964nvahiM6HUVfX3i4gPC/dJWr2xE5uZtYkmNGsUExH/lvSqpK0j4gWyF/FmpeUE4KL0eUfKMhG4WdKlwOfJHvxNjYhaSUskDQIeB44Hrmjs+nnanB8hq9o3lmZm1vZatv/y6cBNabC3l4CTyDpSjJd0MjAPOBwgImZKGk8WvGuAERFRm85zKnA90BW4Ny1FFWtz/hxZo3VXSdvzyR8L3YFuTbxBM7PWUd1ywTkingLqe/FuzwaOHw2Mrid9GjCgKdcuVnPeFziRrH3k0oL0JcD5TbmImVmrKe8X/3Ir1uY8Dhgn6b8i4k+tWCYzs2aLMn8tO69izRrHRsSNwGaSzl55f0RcWk82M7O21UIPBNtasWaNNdLnmq1REDOzFlEZsblos8bV6fNHrVccM7NVVCHNGnnG1viFpO6SOkuaJOlNSce2RuHMzJqsWvmXMpZnXKZ9ImIxcADZmy5bAd8taanMzJqrQsZzzvMSSuf0uT9wS0S8XSkTKJpZBaqQ+JQnON8p6XlgKXBamn37w0bymJm1jY4yTVVEjAR2AXaKiOXAB2TjlpqZlZ+O0qwhqTNwHDAkNWdMAa4qcbnahT5b9OIPvz3j4+3NN+nJTy6dwE0T/s4fxpzJpr3X55X5b3Lsab/m3fc++Pi4jT+/HtMnXcLoX03gsrF3A3Dhd7/OMf81hHXWXoMNvnhSq9+LlcayZR9xzDEj+eij5dTW1rLvvoM544xjAPjDH+7kxhvvplOnKr761S9z7rnZz/355+fywx/+lvff/w9VVVVMmHApq63WpS1vo12JMn/Ql1eeZo0rydqdx6Tt41La/5SqUO3F7JcWMGi/UQBUVYkXp45h4n1P8J0Rw5j88LNcMmYi3zntIL5z2kFc8LNbPs73ix8cx18nP/Wpc93zt+lcNe6vzJjyq9a8BSuxLl06M27caNZYoyvLl9dw9NHnMWTIjnz44UdMmvQ4d955BV26dOatt94FoKamlu9+91Iuvvhs+vbdnHfeWUynTtXFL2KfVuY14rzytM58OSJOiIgH0nIS8OVSF6y92X3wAObOW8i8197kgL135MYJfwfgxgl/58B9Phk35cB9dmLuvEXM+tf8T+Wf+s85/HvRu61ZZGsFklhjja4A1NTUUFNTgyRuueUehg8/jC5dsuft6623DgAPP/xPtt56M/r23RyAHj26U13t4NwkasJSxvIE51pJX6jbkLQFUFvk+A7p8IN2ZfwdjwDQc/21Pw60/170Lhus3x2Abl1X45xTD2T0ZR6qpCOpra1l2LAz2HXX49h11+3Zdtutefnl15k2bSaHH34Oxx47kmee+RcAc+e+hgQnn/wDDjnkTK65xt+VJqtS/qWM5WnW+C7woKSXyH7XbEo2pqklnTtX87W9d+QHP7+16HHfP/swrrjuXj74z7JWKpmVg+rqau6443IWL36fESN+yr/+9Qq1tbUsXvw+48dfwowZsznrrJ8zadK11NbW8uSTs5gw4VK6dl2NE0+8gAEDtmSXXbZt69toPyqkWaNocE7d5t4DBgI9yYLz8xFRNLoUTprYqcdOdFpzy5YpbZnad7fteOrZuSx68z0AFr35Hp/ruQ7/XvQun+u5Dm+8uRiAL2+/JYfsvzOjRx3N2t27sSKCD5ct56pxf23L4lsr6d59TXbe+Uv84x9PsuGG67P33rsiiW222YqqqireeWcxn/vc+gwcOIB1110bgCFDdmLmzBcdnJuiMmJzw80akv4HmEk2ncpTwGYR8XRjgRmySRMjYqeI2KnSAzPA14d90qQBcPf9T3LsYUMAOPawIdx1/5MA7HXYj+g7+Az6Dj6D3/zuXi7+zZ8dmCvc22+/x+LF7wPw4YfLeOSRp9hii97stdcgHnvsaSBryli+vIYePbrzla/swAsvvMzSpR9SU1PLE088y5ZbblzsErayTlX5lzJWrOZ8FtA/It5I7cw3kc2RZQW6rt6FPf7fl/jWqGs/TrtkzERuvPJMTjhiN159/S2O+eZljZ5n9PlHc8SwXenWtQtzHv8Nv7/1QUb/yu2N7d2iRW8zcuRl1NauIGIFQ4d+hd13H8hHHy3n/PMv54ADRtC5cycuuugsJLH22mty4okHc9hhZyOJIUN2Yrfd/Py9KaJCas6KqH+GbknTI2KHhrbz6rrJUY1OAW4dz9J5HuzQ6rPVKofWLYZPyB1zXhp7WNmG8mI1596SLm9oOyLOqCePmVnb6gAPBFceee7JUhbEzKxFlHkXubwam0PQzKx9Ke/nfLnl6edsZtZ+VFdGdK6MuzAzS0LKvTRG0suSZkh6StK0lLaupPslzU6fPQqOHyVpjqQXJO1bkL5jOs8cSZcrx6D4eaapGpwnzcysLFQ1Ycln94jYLiLqBskZCUyKiD7ApLSNpH7AkUB/YCgwRlLdwChXkr2Y1yctQ/PcRmOuyJlmZtb2Sj+2xjCg7pncOODggvRbI2JZRMwF5gADJfUCukfEo5H1Xb6hIE+DGmxzlrQLsCuwgaSzC3Z1BzxMlpmVp5btShfAXyUFcHVEjAU2jIgFABGxQFLPdOxGwGMFeeentOVpfeX0ooo9EOwCrJmOWasgfTFwWGMnNjNrE00YbL9wHKBkbArAdQZHxOspAN+fpuxr8HT1pEWR9KKKdaWbAkyRdH1EvNLYiczMykE0obkiBeKxRfa/nj4XSbqdbBC4hZJ6pVpzL2BROnw+UDgQSm/g9ZTeu570ovK0OV8raZ26DUk9JP0lRz4zs9bXQm3OktaQtFbdOrAP8CzZGEMnpMNOAO5I6xOBIyWtJmlzsgd/U1MTyBJJg1IvjeML8jQoTz/n9SPi3bqNiHinoI3FzKy8tFyb84bA7anXWyfg5oi4T9ITwHhJJwPzgMMBImKmpPHALKAGGBERdROTnApcD3QF7k1LUXmC8wpJm0TEPABJm5KjvcTMrE200NsbEfES8JmBtCPiLWDPBvKMBkbXkz4NGNCU6+cJzt8DHpI0JW0P4dMN6GZm5aMDDHwEQKrG7wAMInvq+O2IeLPkJTMza44yH0Q/r2L9nPtGxPMpMMMnTxc3Sc0c00tfPDOzpsnzWnZ7UKzmfA5wCvDLevYFsEdJSmRmtioqo+JctJ/zKelz99YrjpnZKqr0mrOkQ4tljIj/a/nimJmtokofbB84MH32JBtj44G0vTswGXBwNrPyU+nBOSJOApB0F9CvbqCP9Lrib1uneGZmTRNNGFujnOXp57xZXWBOFgJblag8ZmarptLbnAtMTmNp3ELWS+NI4MGSlsrMrLkqvVmjTkR8S9IhZG8GQjak3u2lLZaZWTNVRmzOPcHrdGBJRPxNUjdJa0XEklIWzMysOaoqpJ9znjkETwEmAFenpI2AP5ewTGZmzVZVlX8pZ3mKNwIYTDYDChExm6x7nZlZ2ZGUeylneZo1lkXER3U3IqkTHjLUzMpUmcfc3PLUnKdIOh/oKmlv4I/AnaUtlplZ80j5l3KWJzifB7wBzAC+AdwDXFDKQpmZNZeq8i/lrGizhqQq4JmIGABc0zpFMjNrvnKvEedVNDhHxApJTxdOU2VmVs6qy7xGnFeeB4K9gJmSpgIf1CVGxEElK5WZWTN1iJpz8qOSl8LMrIWUexe5vIqN57w68E1gS7KHgddFRE1rFczMrDnK/UFfXsVqzuOA5cA/gP2AfsCZrVEoM7PmqpCKc9Hg3C8ivgQg6TpgausUycys+cr9tey8it3G8roVN2eYWXtRpfxLHpKqJf0zTTyCpHUl3S9pdvrsUXDsKElzJL0gad+C9B0lzUj7LleOhvFiwXlbSYvTsgTYpm5d0uJ8t2Vm1rpK8IbgmcBzBdsjgUkR0QeYlLaR1I9svPv+wFBgjKTqlOdKYDjQJy1DG7tog8E5Iqojonta1oqITgXr3XPflplZK2rJ4CypN/A14NqC5GFkz+RInwcXpN8aEcsiYi4wBxiYpvbrHhGPRkQANxTkaVCFtM6YmWVUpfyLNFzStIJl+Eqnuww4F1hRkLZh3dR96bNulM6NgFcLjpuf0jZK6yunF5V3sH0zs3ahKb01ImIsMLb+8+gAYFFEPClptzyXru8SRdKLcnA2s4rSgr01BgMHSdofWB3oLulGYKGkXhGxIDVZLErHzwc2LsjfG3g9pfeuJ70oN2uYWUVpqd4aETEqInpHxGZkD/oeiIhjgYnACemwE4A70vpE4EhJq0nanOzB39TU9LFE0qDUS+P4gjwNcs3ZzCpKK7yEchEwXtLJwDzgcICImClpPDALqAFGRERtynMqcD3QFbg3LUU5OJtZRSnF69sRMRmYnNbfAvZs4LjRwOh60qcBA5pyTQdnM6soHeH1bTOzdqfiR6UzM2uPKmVsDQdnM6soFVJxLn1wnvHs0aW+hLVDH9a+3dZFsDK0enXjxzQm74BG5c41ZzOrKA7OZmZlqEqNvhndLjg4m1lF6eSas5lZ+XHN2cysDLnN2cysDFVIN2cHZzOrLK45m5mVIbnN2cys/HSI3hppYOiBZPNdBdno/VPTJIVmZmWn4ntrSNoHGAPMBl5Lyb2BLSWdFhF/bYXymZk1SUdoc/41sFdEvFyYmKZfuQf4YgnLZWbWLB2ht0YnPj2dd53XgM6lKY6Z2arpCDXn3wFPSLoVeDWlbUw20eF1pS6YmVlzVHybc0T8TNKfgWHALoDIatLHRMSs1imemVnTdIjeGhHxHPBcK5XFzGyVVUrNOVfbuaQLi22bmZWLKuVfylnel1CebGTbzKwslHvQzStXcI6IO4ttm5mVi0rpStfofUjaStIkSc+m7W0kXVD6opmZNV2nqsi9FCNpdUlTJT0taaakH6X0dSXdL2l2+uxRkGeUpDmSXpC0b0H6jpJmpH2Xp7evi8rzS+YaYBSwHCAiniHrTmdmVnaqmrA0YhmwR0RsC2wHDJU0CBgJTIqIPsCktI2kfmSxsT8wFBgjqW7K2iuB4UCftAzNcx+N6RYRU1dKq8mRz8ys1bXUA8HIvJ82O6clyLoXj0vp44CD0/ow4NaIWBYRc4E5wEBJvYDuEfFoGpfohoI8Dd9Hjnt9U9IXUqGQdBiwIEc+M7NWJ0UTFg2XNK1gGf7pc6la0lPAIuD+iHgc2DAiFgCkz57p8I345IU9yN4L2Sgt8+tJLyrPA8ERwFigr6TXgLnAMTnymZm1uqb01oiIsWTxraH9tcB2ktYBbpc0oMjp6rtyFEkvKk9wfiUi9pK0BlAVEUty5DEzaxOl6K0REe9KmkzWVrxQUq+IWJCaLBalw+aTDXFRpzfZMMvz0/rK6UXluY+5ksYCg4D3GzvYzKwttWBvjQ1SjRlJXYG9gOeBicAJ6bATgDvS+kTgSEmrpdE7+5CNf78AWCJpUOqlcXxBnobvI8e9bg0cSNa8cZ2ku8gavR/KkdfMrFW14EsovYBxqcdFFTA+Iu6S9CgwXtLJwDzgcICImClpPDCLrNPEiNQsAnAqcD3QFbg3LUU1GpwjYikwPhWmB9k4z1OA6qIZzczaQEsFptRtePt60t8C9mwgz2hgdD3p04Bi7dWfkXdsja9KGgNMB1YHvt6Ui5iZtZYqRe6lnDVac5Y0F3iKrPb83Yj4oNSFMjNrro40tsa2EbG45CUxM2sBFR+cJZ0bEb8ARkufrf9HxBklLZmZWTN0rpCRj4rVnOsG2Z/WGgUxM2sJ5d6WnFexaarqhgX9T0T8sXCfpMNLWiozs2aqlGaNPH8AjMqZZmbW5qqbsJSzYm3O+wH7AxtJurxgV3c8Kp2ZlalKqTkXa3N+nay9+SA+PS3VEuDbpSyUmVlzdW7ktez2olib89PA05JujojlrVgmM7Nm6wg15zqbSfoZ0I/s7UAAImKLkpXKzKyZKiU453kg+HuyKVZqgN3JRvH/QykLZWbWXC01E0pbyxOcu0bEJEAR8UpEXAjsUdpimZk1T7Ui91LO8jRrfCipCpgt6VvAa3wyLYuZWVmpkBcEcwXns4BuwBnAT8hqzScUy2Bm1lY6VUh0zjOe8xNp9X3gpNIWx8xs1ZR7c0VeeYYMvZPPTkb4Hlkf6Ksj4sNSFMzMrDnK/UFfXnn+AHiJrNZ8TVoWAwuBrdK2mVnZqJTeGnnanLePiCEF23dK+ntEDJE0s1QFMzNrjnIPunnlCc4bSNokIuYBSNoEWD/t+6hkJTMza4aKf327wDnAQ5JeBARsDpwmaQ1gXCkLZ2bWVBXSWSNXb417JPUB+pIF5+cLHgJeVsKytQuX/fg2pj40i3V6rMmY2777cfrE2x7irvEPU11dxZe/8kX++4wDWPzuB/x05A3MnvUqex2wE6eee+jHx3//9Gt4563F1NasoP/2m3PquYdSXV0pX7OO6+W5Czj37DEfb8+fv4jTTj+UY4/fF4Bxv7uHSy+5jckP/4YePdZi+Uc1/PjC3zNr5stUVYlzRx3Dlwd+sa2K3y51mGYNSd2As4FNI+IUSX0kbR0Rd5W+eOVvrwN24oCvD+bSH97ycdrT0+bw2JSZ/PaWc+jcpRPvvr0EgC6rdeK4bw7llRcX8MqL//7UeUb97Di6rbk6EcFPz7uBhyY9zVf3+cys7NbObLZ5L8bf/hMAamtXsPduZ7HHnjsC8O8Fb/HoozPp1Wu9j4//04TJ2ecdo3nrrcWM+MYl3Dz+Qqqq/Is6r+oKCc55x9b4CNglbc8H/rdkJWpnBuzwBdbq3u1Taff86REOP2F3OnfJfvets+5aAKzedTX6b7c5nbt0/sx5uq2ZjSlVW7uCmuU1iAr5htnHHn9sJhtvsgGf3yh7ZHPxz2/m2+ccgfTJz/qlF19n50H9AFhvve6stdYazHx2bpuUt72qUuReipG0saQHJT0naaakM1P6upLulzQ7ffYoyDNK0hxJL0jatyB9R0kz0r7LVfhDb+g+ctzrF9JEr8sBImIpOHIU89orbzLzqbl8+8Rfc97wMfxr5rxc+b5/+liO3udCuq6xOoP33KbEpbTWdt89jzN0/0EATH5gOj179mDrvpt86pittt6YyQ/8k5qaWubPf4PnZr3Mwn+/3RbFbbdasCtdDXBORHwRGASMkNQPGAlMiog+wKS0Tdp3JNAfGAqMkVQ34cqVwHCgT1qGNnofOe71I0ldSS+iSPoCsKxYBknDJU2TNO3W39+X4xKVZUVtLe8vWcqlvz+D/z7zAC46/w9ENP4E+SdXDOfGe3/A8o9qeGbanFYoqbWW5R/VMOXBf7LPvgNZunQZ11x9J6edfuhnjjv40CFs+LkeHH34hVz8s5vYdrstqa4u9wmVyksn5V+KiYgFETE9rS8hm/R6I2AYn3SGGAccnNaHAbdGxLKImAvMAQZK6gV0j4hHIwsENxTkafg+ctzrD4H7gI0l3QQMBk5s5KbGAmMB5iy+szL6tTTBej3XYdfdByCJrftvglTF4nc/YO0eazaat8tqndl5SD8em/Is2++8VSuU1lrDQ/94hr79NmW99ddm9r9e5bXX3uDrh3wfgIUL3+bI//oBN932Q9bfYB2+O/KYj/Mdf/RP2GTTDduq2O1S4w0GzTmnNgO2Bx4HNoyIBZAFcEl1A8FtBDxWkG1+Slue1ldOLypPb437JU0nq9YLODMi3mz0bjqwXXbrz9NPzGGbHbfktVfeoGZ5Dd3XWaPB45f+ZxlL/7OMddfvTm1NLdMefp7+22/eiiW2Urv3nsfYLzVp9NlqYyY/9JuP9+231znc/McL6dFjLZYuXUYEdOu2Go8+8izV1VV8YctG/x1bgabEZknDyZob6oxNlcvCY9YE/gScFRGLizQX17cjiqQXVWyC101WSpqRPrsVvpTS0f38ezcy48kXWfzuBxz/tZ9wzPB92PuggVz24/GcdsTFdOrcibMvPPLjhz4nHTSa/3zwITXLa3l0ykz+94pTWGvtNfjx2b9j+fJaVtSuYJsvb8n+h+7SyJWtvVi6dBmPPfIs37/wxEaPffvtxZx6yiVUVYmePXsw+qJvlL6AFaYpNefCv/LrP5c6kwXmmyLi/1LyQkm9Uq25F7Aopc8HNi7I3ptsLtb5aX3l9KLUUFuopBl8NuoHsAHQMyJyNYR1xGYNa1zvNTZo6yJYGVq9etAqN0pMf/Pu3DFnh/W/1nA1OKtRjQPejoizCtIvBt6KiIskjQTWjYhzJfUHbgYGAp8ne1jYJyJqJT0BnE7WLHIPcEVE3FOsbMUmeP3SSgXdDDgP2Av4abGTmpm1FbXckKGDgeOAGZKeSmnnAxcB4yWdDMwDDgeIiJmSxgOzyHp6jIiI2pTvVOB6oCtwb1qKyvMSSh/ge8DOwC+BMzwbt5mVq5Z6QzAiHqLhJuw9G8gzGhhdT/o0YEBTrl+szXkAWVDuD/wCOLngt4CZWVmqlJcwitWcnwZeBe4ma0MZWPiUMiLOKG3RzMyariOMrfHfrVYKM7MWUiGxuegDQQ8HambtTileQmkLed4QNDNrNypl/D4HZzOrKJXS5tzoLxlJg/OkmZmVAzVhKWd5/gK4ImeamVmbkyL3Us6K9XPeBdiVbILXswt2dQc8hqGZlaVyrxHnVazNuQuwZjpmrYL0xcBhpSyUmVlzVXxvjYiYAkyRdH1EvNKKZTIza7aONIfgtZLWqduQ1EPSX0pXJDOz5quUB4J5utKtHxHv1m1ExDsFI/+bmZWVSmnWyFNzXlE48L6kTckxir+ZWVvoSDXn7wEPSZqStofw6WldzMzKRqW8hJJnDsH7JO3AJ3MIfttzCJpZuaqQ2Fy0n3PfiHg+BWb4ZM6rTdIcgtNLXzwzs6apKvOXS/IqVnM+BziFbPaTlQWwR0lKZGa2CirlgWCxfs6npM/dW684ZmarpkJic9FmjUOLZSyYJtzMrGx0hCFDD0yfPcnG2Hggbe8OTAYcnM2s7HSEZo2TACTdBfSLiAVpuxfw29YpnplZ06hC6s55+jlvVheYk4XAViUqj5nZKpE6TnCenMbSuIWsl8aRwIMlLZWZWbNVRrtGo79iIuJbwFXAtsB2wNiIOL3E5TIzaxY14b9GzyX9TtIiSc8WpK0r6X5Js9Nnj4J9oyTNkfSCpH0L0neUNCPtu1xqvGU8b/1/OnB3RHwb+IuktRrLYGbWNlp0dI3rgaErpY0EJkVEH2BS2kZSP7KWhf4pzxhJdROTXEk27EWftKx8zs/IM4fgKcAE4OqUtBHw58bymZm1Bakq99KYiPg78PZKycOAcWl9HHBwQfqtEbEsIuYCc4CBqRNF94h4NCICuKEgT4Py1JxHAIPJZkAhImaTda8zMys7oir/Ig2XNK1gyTOo24Z1nSTSZ1083Ah4teC4+Slto7S+cnpReR4ILouIj+qaSCR1wkOGmlmZytOWXCcixgJjW+zS9VyiSHpReWrOUySdD3SVtDfwR+DOHPnMzNpAVROWZlmYmirq3vtYlNLnAxsXHNebbMC4+Wl95fRG76Ix5wFvADOAbwD3ABfkyGdm1uok5V6aaSJwQlo/AbijIP1ISatJ2pzswd/U1PSxRNKg1Evj+II8DSrarKGsxfyZiBgAXNO8+zAza00t189Z0i3AbsD6kuYDPwQuAsZLOhmYBxwOEBEzJY0HZgE1wIiIqE2nOpWs50dX4N60FFU0OEfECklPp/Gb5zXj3szMWlVT2pwbExFHNbBrzwaOHw2Mrid9GjCgKdfO80CwFzBT0lTgg4KLHdSUC5mZtQZR3fhB7UCe4PyjkpfCzKyFrEJbclkpNp7z6sA3gS3JHgZeFxE1rVUwM7PmqfDgTPbmy3LgH8B+QD/gzNYolJlZc3WEIUP7RcSXACRdB0xtnSKZma2Kyq85L69biYiaSmnHMbPK1hHGc95W0uK0LrI3BBen9YiI7iUvnZlZE1V8s0ZEVEZ/FDPrYCrjr/w8XenMzNqNlnwJpS05OJtZRamU52MOzmZWYSq8zdnMrD2q+AeCZmbtkZs1zMzKkmvOZmZlp1J6ayibDNZag6Thac4ys4/5e2H1qYz6f/uRZ2Zf63j8vbDPcHA2MytDDs5mZmXIwbl1uV3R6uPvhX2GHwiamZUh15zNzMpQxQVnSYdICkl9cxx7lqRuq3CtEyX9poH0NyQ9JWmWpFOace5vSjq+4HyfL9h3raR+zS13wXkOlzRT0gpJO63q+cpNGX0XVkjapiDtWUmbNfdaDVx/O0n7F2wfJGlkC517lKQ5kl6QtG9LnNMaV3HBGTgKeAg4MsexZwHN/gfZiNsiYjtgN+CnkjZsSuaIuCoibkibJwKfL9j3PxExqwXK+CxwKPD3FjhXOSqX78J84HslOned7YCPg3NETIyIi1b1pKkScCTQHxgKjJHksd5bQUUFZ0lrAoOBkyn4BympWtIlkmZIekbS6ZLOIAt4D0p6MB33fkGewyRdn9YPlPS4pH9K+ltTAm1ELAJeBDaVtGc6xwxJv5O0Wjr/RamG/YykS1LahZK+I+kwYCfgplQT7yppsqSdJJ0q6RcFZT5R0hVp/VhJU1Oeq+v7BxURz0XEC3nvpT0ps+/CXUB/SVvXU859JD0qabqkP6ZyI2l/Sc9LekjS5ZLuSukDJT2Srv+IpK0ldQF+DByRft5H1NXkJa0t6WWluZskdZP0qqTOkr4g6T5JT0r6RwN/YQwDbo2IZRExF5gDDMxxz7aKKio4AwcD90XEv4C3Je2Q0ocDmwPbR8Q2wE0RcTnwOrB7ROzeyHkfAgZFxPbArcC5eQskaQtgC7La0/XAEWni3E7AqZLWBQ4B+qey/W9h/oiYAEwDjomI7SJiacHuCWQ13zpHALdJ+mJaH5xq77XAMak811ZiE0Y9DqZ8vgsrgF8A5xcmSlofuADYKyJ2IPs5ny1pdeBqYL+I+AqwQUG254Eh6fo/AH4aER+l9dvSd+S2uoMj4j3gaeCrKelA4C8RsZysl8jpEbEj8B1gTCrXQZJ+nI7fCHi14PrzU5qVWKWNrXEUcFlavzVtTwf2Aq6KiBqAiHi7ieftTRb0egFdgLk58hwh6SvAMuAbZP/A5qZgATAOGAH8BvgQuFbS3WS1rFwi4g1JL0kaBMwGtgYeTufdEXhC2QhdXYFFKc//5D1/O1dO3wWAm4HvSdq8IG0Q0A94OP2cugCPAn2Bl1JNFeAWPnmLcG1gnKQ+QACdc1z7NrJf1g+S/RUxJtXQdwX+qE9GcVsNsiYRYGJKq2+gCnfxagUVE5wlrQfsAQyQFEA1EJLOJU1Km+M0hcesXrB+BXBpREyUtBtwYY5z3RYR3yoo33b1XjCb2XwgsCfZP5xvpfvI6zbg62Q1qtsjIpT9axsXEaOacJ6KUYbfhbqf8y+B8wqLCtwfEUetVP7ti5zqJ8CDEXGIsoeKk3NcfiLws/RX2o7AA8AawLvpL6ti5gMbF2z3Jvsrw0qskpo1DgNuiIhNI2KziNiYrFbzFeCvwDcldQJIX1KAJcBaBedYKOmLqX3ukIL0tYHX0voJzSzf88BmkrZM28cBU1INZu2IuIfsodR29eRduZyF/o/sT/ijyAI1wCTgMEk9IbtfSZs2s9ztUbl+F64nq7nXNVM8Bgyu+06k9uCtyL4rW+iTHh1HNHD9EwvSG/yORMT7wFTg18BdEVEbEYuBuZIOT9eWpG3ryT4ROFLSaqnW3yedy0qskoLzUcDtK6X9CTgauBaYBzwj6emUBlmb2711D4GAkWTNCg8ACwrOcyHZn3//AN5sTuEi4kPgpHSeGWTtkFeR/YO6S9IzwBTg2/Vkvx64Kj3s6brSed8BZgGbRsTUlDaLrC3zr+m89wO94NNtzsq6ms0HdgHulvSX5txbGSrL70JqG74c6Jm23yALsLekn9NjQN/0XOE04D5JDwELgffSaX5BVgt+mOwvgjoPAv3qHgjWc/nbgGP55Bc4ZM8hTk7/H2aSPfz7VJtzRMwExpN9x+4DRkREbVPu25rHbwialSFJa0bE+6mJ6rfA7Ij4VVuXy1pPJdWczSrJKZKeIqvRrk3We8M6ENeczczKkGvOZmZlyMHZzKwMOTibmZUhB2czszLk4GxmVoYcnM3MytD/B/tK5vRktkFjAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize confusion matrix with seaborn heatmap\n",
    "\n",
    "cm_matrix = pd.DataFrame(data=cm, columns=['Actual Positive:1', 'Actual Negative:0'], \n",
    "                                 index=['Predict Positive:1', 'Predict Negative:0'])\n",
    "\n",
    "sns.heatmap(cm_matrix, annot=True, fmt='d', cmap='YlGnBu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **16. Classification metrices** <a class=\"anchor\" id=\"16\"></a>\n",
    "\n",
    "[Table of Contents](#0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification Report\n",
    "\n",
    "\n",
    "**Classification report** is another way to evaluate the classification model performance. It displays the  **precision**, **recall**, **f1** and **support** scores for the model. I have described these terms in later.\n",
    "\n",
    "We can print a classification report as follows:-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "       <=50K       0.81      0.95      0.88      7407\n",
      "        >50K       0.67      0.32      0.43      2362\n",
      "\n",
      "    accuracy                           0.80      9769\n",
      "   macro avg       0.74      0.63      0.65      9769\n",
      "weighted avg       0.78      0.80      0.77      9769\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Keep it Up !! All the Best .."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}