Classify Song Genres from Audio Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Preparing our dataset\n",
    "These recommendations are so on point! How does this playlist know me so well?\n",
    "\n",
    "Project Image Record\n",
    "\n",
    "Over the past few years, streaming services with huge catalogs have become the primary means through which most people listen to their favorite music. But at the same time, the sheer amount of music on offer can mean users might be a bit overwhelmed when trying to look for newer music that suits their tastes.\n",
    "\n",
    "For this reason, streaming services have looked into means of categorizing music to allow for personalized recommendations. One method involves direct analysis of the raw audio information in a given song, scoring the raw data on a variety of metrics. Today, we'll be examining data compiled by a research group known as The Echo Nest. Our goal is to look through this dataset and classify songs as being either 'Hip-Hop' or 'Rock' - all without listening to a single one ourselves. In doing so, we will learn how to clean our data, do some exploratory data visualization, and use feature reduction towards the goal of feeding our data through some simple machine learning algorithms, such as decision trees and logistic regression.\n",
    "\n",
    "To begin with, let's load the metadata about our tracks alongside the track metrics compiled by The Echo Nest. A song is about more than its title, artist, and number of listens. We have another dataset that has musical features of each track such as danceability and acousticness on a scale from -1 to 1. These exist in two different files, which are in different formats - CSV and JSON. While CSV is a popular file format for denoting tabular data, JSON is another common file format in which databases often return the results of a given query.\n",
    "\n",
    "Let's start by creating two pandas DataFrames out of these files that we can merge so we have features and labels (often also referred to as X and y) for the classification later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 4802 entries, 0 to 4801\n",
      "Data columns (total 10 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   track_id          4802 non-null   int64  \n",
      " 1   acousticness      4802 non-null   float64\n",
      " 2   danceability      4802 non-null   float64\n",
      " 3   energy            4802 non-null   float64\n",
      " 4   instrumentalness  4802 non-null   float64\n",
      " 5   liveness          4802 non-null   float64\n",
      " 6   speechiness       4802 non-null   float64\n",
      " 7   tempo             4802 non-null   float64\n",
      " 8   valence           4802 non-null   float64\n",
      " 9   genre_top         4802 non-null   object \n",
      "dtypes: float64(8), int64(1), object(1)\n",
      "memory usage: 412.7+ KB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>track_id</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>liveness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>valence</th>\n",
       "      <th>genre_top</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>0.416675</td>\n",
       "      <td>0.675894</td>\n",
       "      <td>0.634476</td>\n",
       "      <td>0.010628</td>\n",
       "      <td>0.177647</td>\n",
       "      <td>0.159310</td>\n",
       "      <td>165.922</td>\n",
       "      <td>0.576661</td>\n",
       "      <td>Hip-Hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>0.374408</td>\n",
       "      <td>0.528643</td>\n",
       "      <td>0.817461</td>\n",
       "      <td>0.001851</td>\n",
       "      <td>0.105880</td>\n",
       "      <td>0.461818</td>\n",
       "      <td>126.957</td>\n",
       "      <td>0.269240</td>\n",
       "      <td>Hip-Hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>0.043567</td>\n",
       "      <td>0.745566</td>\n",
       "      <td>0.701470</td>\n",
       "      <td>0.000697</td>\n",
       "      <td>0.373143</td>\n",
       "      <td>0.124595</td>\n",
       "      <td>100.260</td>\n",
       "      <td>0.621661</td>\n",
       "      <td>Hip-Hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>134</td>\n",
       "      <td>0.452217</td>\n",
       "      <td>0.513238</td>\n",
       "      <td>0.560410</td>\n",
       "      <td>0.019443</td>\n",
       "      <td>0.096567</td>\n",
       "      <td>0.525519</td>\n",
       "      <td>114.290</td>\n",
       "      <td>0.894072</td>\n",
       "      <td>Hip-Hop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>153</td>\n",
       "      <td>0.988306</td>\n",
       "      <td>0.255661</td>\n",
       "      <td>0.979774</td>\n",
       "      <td>0.973006</td>\n",
       "      <td>0.121342</td>\n",
       "      <td>0.051740</td>\n",
       "      <td>90.241</td>\n",
       "      <td>0.034018</td>\n",
       "      <td>Rock</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   track_id  acousticness  danceability    energy  instrumentalness  liveness  \\\n",
       "0         2      0.416675      0.675894  0.634476          0.010628  0.177647   \n",
       "1         3      0.374408      0.528643  0.817461          0.001851  0.105880   \n",
       "2         5      0.043567      0.745566  0.701470          0.000697  0.373143   \n",
       "3       134      0.452217      0.513238  0.560410          0.019443  0.096567   \n",
       "4       153      0.988306      0.255661  0.979774          0.973006  0.121342   \n",
       "\n",
       "   speechiness    tempo   valence genre_top  \n",
       "0     0.159310  165.922  0.576661   Hip-Hop  \n",
       "1     0.461818  126.957  0.269240   Hip-Hop  \n",
       "2     0.124595  100.260  0.621661   Hip-Hop  \n",
       "3     0.525519  114.290  0.894072   Hip-Hop  \n",
       "4     0.051740   90.241  0.034018      Rock  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# Read in track metadata with genre labels\n",
    "tracks = pd.read_csv(\"datasets/Classifying songs/fma-rock-vs-hiphop.csv\")\n",
    "\n",
    "# Read in track metrics with the features\n",
    "echonest_metrics = pd.read_json(\"datasets/Classifying songs/echonest-metrics.json\", precise_float=True)\n",
    "\n",
    "# Merge the relevant columns of tracks and echonest_metrics\n",
    "echo_tracks = echonest_metrics.merge(tracks[[\"track_id\", \"genre_top\"]], on = \"track_id\")\n",
    "\n",
    "# Inspect the resultant dataframe\n",
    "echo_tracks.info()\n",
    "\n",
    "echo_tracks.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Pairwise relationships between continuous variables\n",
    "<p>We typically want to avoid using variables that have strong correlations with each other -- hence avoiding feature redundancy -- for a few reasons:</p>\n",
    "<ul>\n",
    "<li>To keep the model simple and improve interpretability (with many features, we run the risk of overfitting).</li>\n",
    "<li>When our datasets are very large, using fewer features can drastically speed up our computation time.</li>\n",
    "</ul>\n",
    "<p>To get a sense of whether there are any strongly correlated features in our data, we will use built-in functions in the <code>pandas</code> package.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style  type=\"text/css\" >\n",
       "    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col0 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col1 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col2 {\n",
       "            background-color:  #d2d2e7;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col3 {\n",
       "            background-color:  #b5c4df;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col4 {\n",
       "            background-color:  #f5eef6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col5 {\n",
       "            background-color:  #e9e5f1;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col6 {\n",
       "            background-color:  #d1d2e6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col7 {\n",
       "            background-color:  #e1dfed;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col8 {\n",
       "            background-color:  #dedcec;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col0 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col1 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col2 {\n",
       "            background-color:  #e0dded;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col3 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col4 {\n",
       "            background-color:  #97b7d7;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col5 {\n",
       "            background-color:  #f3edf5;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col6 {\n",
       "            background-color:  #b8c6e0;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col7 {\n",
       "            background-color:  #e1dfed;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col8 {\n",
       "            background-color:  #e2dfee;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col0 {\n",
       "            background-color:  #bdc8e1;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col1 {\n",
       "            background-color:  #d0d1e6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col2 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col3 {\n",
       "            background-color:  #fbf3f9;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col4 {\n",
       "            background-color:  #f3edf5;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col5 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col6 {\n",
       "            background-color:  #80aed2;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col7 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col8 {\n",
       "            background-color:  #529bc7;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col0 {\n",
       "            background-color:  #a7bddb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col1 {\n",
       "            background-color:  #f5eff6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col2 {\n",
       "            background-color:  #fef6fa;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col3 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col4 {\n",
       "            background-color:  #c4cbe3;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col5 {\n",
       "            background-color:  #dcdaeb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col6 {\n",
       "            background-color:  #dedcec;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col7 {\n",
       "            background-color:  #adc1dd;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col8 {\n",
       "            background-color:  #d9d8ea;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col0 {\n",
       "            background-color:  #f4eef6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col1 {\n",
       "            background-color:  #97b7d7;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col2 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col3 {\n",
       "            background-color:  #d2d3e7;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col4 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col5 {\n",
       "            background-color:  #fdf5fa;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col6 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col7 {\n",
       "            background-color:  #d9d8ea;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col8 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col0 {\n",
       "            background-color:  #bdc8e1;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col1 {\n",
       "            background-color:  #ced0e6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col2 {\n",
       "            background-color:  #ede8f3;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col3 {\n",
       "            background-color:  #bdc8e1;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col4 {\n",
       "            background-color:  #dbdaeb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col5 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col6 {\n",
       "            background-color:  #c0c9e2;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col7 {\n",
       "            background-color:  #dcdaeb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col8 {\n",
       "            background-color:  #e8e4f0;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col0 {\n",
       "            background-color:  #d0d1e6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col1 {\n",
       "            background-color:  #b8c6e0;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col2 {\n",
       "            background-color:  #93b5d6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col3 {\n",
       "            background-color:  #eae6f1;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col4 {\n",
       "            background-color:  #fff7fb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col5 {\n",
       "            background-color:  #eae6f1;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col6 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col7 {\n",
       "            background-color:  #dbdaeb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col8 {\n",
       "            background-color:  #bfc9e1;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col0 {\n",
       "            background-color:  #d0d1e6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col1 {\n",
       "            background-color:  #d0d1e6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col2 {\n",
       "            background-color:  #fef6fa;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col3 {\n",
       "            background-color:  #a7bddb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col4 {\n",
       "            background-color:  #c5cce3;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col5 {\n",
       "            background-color:  #f0eaf4;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col6 {\n",
       "            background-color:  #c8cde4;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col7 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col8 {\n",
       "            background-color:  #d6d6e9;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col0 {\n",
       "            background-color:  #c6cce3;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col1 {\n",
       "            background-color:  #cdd0e5;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col2 {\n",
       "            background-color:  #4c99c5;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col3 {\n",
       "            background-color:  #d1d2e6;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col4 {\n",
       "            background-color:  #efe9f3;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col5 {\n",
       "            background-color:  #f7f0f7;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col6 {\n",
       "            background-color:  #a5bddb;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col7 {\n",
       "            background-color:  #d3d4e7;\n",
       "            color:  #000000;\n",
       "        }    #T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col8 {\n",
       "            background-color:  #023858;\n",
       "            color:  #f1f1f1;\n",
       "        }</style><table id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253\" ><thead>    <tr>        <th class=\"blank level0\" ></th>        <th class=\"col_heading level0 col0\" >track_id</th>        <th class=\"col_heading level0 col1\" >acousticness</th>        <th class=\"col_heading level0 col2\" >danceability</th>        <th class=\"col_heading level0 col3\" >energy</th>        <th class=\"col_heading level0 col4\" >instrumentalness</th>        <th class=\"col_heading level0 col5\" >liveness</th>        <th class=\"col_heading level0 col6\" >speechiness</th>        <th class=\"col_heading level0 col7\" >tempo</th>        <th class=\"col_heading level0 col8\" >valence</th>    </tr></thead><tbody>\n",
       "                <tr>\n",
       "                        <th id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253level0_row0\" class=\"row_heading level0 row0\" >track_id</th>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col0\" class=\"data row0 col0\" >1.000000</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col1\" class=\"data row0 col1\" >-0.372282</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col2\" class=\"data row0 col2\" >0.049454</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col3\" class=\"data row0 col3\" >0.140703</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col4\" class=\"data row0 col4\" >-0.275623</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col5\" class=\"data row0 col5\" >0.048231</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col6\" class=\"data row0 col6\" >-0.026995</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col7\" class=\"data row0 col7\" >-0.025392</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row0_col8\" class=\"data row0 col8\" >0.010070</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253level0_row1\" class=\"row_heading level0 row1\" >acousticness</th>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col0\" class=\"data row1 col0\" >-0.372282</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col1\" class=\"data row1 col1\" >1.000000</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col2\" class=\"data row1 col2\" >-0.028954</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col3\" class=\"data row1 col3\" >-0.281619</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col4\" class=\"data row1 col4\" >0.194780</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col5\" class=\"data row1 col5\" >-0.019991</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col6\" class=\"data row1 col6\" >0.072204</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col7\" class=\"data row1 col7\" >-0.026310</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row1_col8\" class=\"data row1 col8\" >-0.013841</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253level0_row2\" class=\"row_heading level0 row2\" >danceability</th>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col0\" class=\"data row2 col0\" >0.049454</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col1\" class=\"data row2 col1\" >-0.028954</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col2\" class=\"data row2 col2\" >1.000000</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col3\" class=\"data row2 col3\" >-0.242032</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col4\" class=\"data row2 col4\" >-0.255217</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col5\" class=\"data row2 col5\" >-0.106584</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col6\" class=\"data row2 col6\" >0.276206</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col7\" class=\"data row2 col7\" >-0.242089</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row2_col8\" class=\"data row2 col8\" >0.473165</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253level0_row3\" class=\"row_heading level0 row3\" >energy</th>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col0\" class=\"data row3 col0\" >0.140703</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col1\" class=\"data row3 col1\" >-0.281619</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col2\" class=\"data row3 col2\" >-0.242032</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col3\" class=\"data row3 col3\" >1.000000</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col4\" class=\"data row3 col4\" >0.028238</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col5\" class=\"data row3 col5\" >0.113331</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col6\" class=\"data row3 col6\" >-0.109983</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col7\" class=\"data row3 col7\" >0.195227</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row3_col8\" class=\"data row3 col8\" >0.038603</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253level0_row4\" class=\"row_heading level0 row4\" >instrumentalness</th>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col0\" class=\"data row4 col0\" >-0.275623</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col1\" class=\"data row4 col1\" >0.194780</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col2\" class=\"data row4 col2\" >-0.255217</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col3\" class=\"data row4 col3\" >0.028238</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col4\" class=\"data row4 col4\" >1.000000</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col5\" class=\"data row4 col5\" >-0.091022</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col6\" class=\"data row4 col6\" >-0.366762</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col7\" class=\"data row4 col7\" >0.022215</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row4_col8\" class=\"data row4 col8\" >-0.219967</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253level0_row5\" class=\"row_heading level0 row5\" >liveness</th>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col0\" class=\"data row5 col0\" >0.048231</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col1\" class=\"data row5 col1\" >-0.019991</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col2\" class=\"data row5 col2\" >-0.106584</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col3\" class=\"data row5 col3\" >0.113331</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col4\" class=\"data row5 col4\" >-0.091022</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col5\" class=\"data row5 col5\" >1.000000</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col6\" class=\"data row5 col6\" >0.041173</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col7\" class=\"data row5 col7\" >0.002732</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row5_col8\" class=\"data row5 col8\" >-0.045093</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253level0_row6\" class=\"row_heading level0 row6\" >speechiness</th>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col0\" class=\"data row6 col0\" >-0.026995</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col1\" class=\"data row6 col1\" >0.072204</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col2\" class=\"data row6 col2\" >0.276206</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col3\" class=\"data row6 col3\" >-0.109983</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col4\" class=\"data row6 col4\" >-0.366762</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col5\" class=\"data row6 col5\" >0.041173</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col6\" class=\"data row6 col6\" >1.000000</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col7\" class=\"data row6 col7\" >0.008241</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row6_col8\" class=\"data row6 col8\" >0.149894</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253level0_row7\" class=\"row_heading level0 row7\" >tempo</th>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col0\" class=\"data row7 col0\" >-0.025392</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col1\" class=\"data row7 col1\" >-0.026310</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col2\" class=\"data row7 col2\" >-0.242089</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col3\" class=\"data row7 col3\" >0.195227</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col4\" class=\"data row7 col4\" >0.022215</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col5\" class=\"data row7 col5\" >0.002732</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col6\" class=\"data row7 col6\" >0.008241</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col7\" class=\"data row7 col7\" >1.000000</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row7_col8\" class=\"data row7 col8\" >0.052221</td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                        <th id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253level0_row8\" class=\"row_heading level0 row8\" >valence</th>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col0\" class=\"data row8 col0\" >0.010070</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col1\" class=\"data row8 col1\" >-0.013841</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col2\" class=\"data row8 col2\" >0.473165</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col3\" class=\"data row8 col3\" >0.038603</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col4\" class=\"data row8 col4\" >-0.219967</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col5\" class=\"data row8 col5\" >-0.045093</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col6\" class=\"data row8 col6\" >0.149894</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col7\" class=\"data row8 col7\" >0.052221</td>\n",
       "                        <td id=\"T_9d5900be_8d48_11ed_84f9_6cc2176c0253row8_col8\" class=\"data row8 col8\" >1.000000</td>\n",
       "            </tr>\n",
       "    </tbody></table>"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x271dc17d910>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a correlation matrix\n",
    "corr_metrics = echo_tracks.corr()\n",
    "corr_metrics.style.background_gradient()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Splitting our data\n",
    "<p>As mentioned earlier, it can be particularly useful to simplify our models and use as few features as necessary to achieve the best result. Since we didn't find any particularly strong correlations between our features, we can now split our data into an array containing our features, and another containing the labels - the genre of the track. </p>\n",
    "<p>Once we have split the data into these arrays, we will perform some preprocessing steps to optimize our model development.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[4.16675233e-01 6.75893985e-01 6.34476268e-01 ... 1.59310065e-01\n",
      "  1.65922000e+02 5.76660988e-01]\n",
      " [3.74407769e-01 5.28643062e-01 8.17461132e-01 ... 4.61818128e-01\n",
      "  1.26957000e+02 2.69240242e-01]\n",
      " [4.35668989e-02 7.45565870e-01 7.01469992e-01 ... 1.24595342e-01\n",
      "  1.00260000e+02 6.21661224e-01]\n",
      " ...\n",
      " [1.04777022e-02 6.52483062e-01 6.57497709e-01 ... 2.29174375e-01\n",
      "  9.48850000e+01 4.32240157e-01]\n",
      " [6.79063452e-02 4.32421004e-01 7.64508334e-01 ... 3.10552525e-01\n",
      "  1.71329000e+02 5.80087111e-01]\n",
      " [1.53518164e-01 6.38659813e-01 7.62567227e-01 ... 3.03372337e-01\n",
      "  7.78420000e+01 6.56612274e-01]] \n",
      "\n",
      "0       Hip-Hop\n",
      "1       Hip-Hop\n",
      "2       Hip-Hop\n",
      "3       Hip-Hop\n",
      "4          Rock\n",
      "         ...   \n",
      "4797    Hip-Hop\n",
      "4798    Hip-Hop\n",
      "4799    Hip-Hop\n",
      "4800    Hip-Hop\n",
      "4801    Hip-Hop\n",
      "Name: genre_top, Length: 4802, dtype: object \n",
      "\n",
      "[[9.48677459e-01 1.37199242e-01 2.47403596e-02 ... 4.94376494e-02\n",
      "  2.11283000e+02 3.81535017e-02]\n",
      " [1.02727000e-05 1.18115308e-01 7.50281301e-01 ... 4.45210657e-02\n",
      "  9.51060000e+01 3.59604392e-01]\n",
      " [3.12262570e-02 6.28698289e-01 8.31143855e-01 ... 6.79014582e-02\n",
      "  9.70140000e+01 6.53131979e-01]\n",
      " ...\n",
      " [6.99944191e-01 4.96842451e-01 8.91126340e-01 ... 1.17408716e-01\n",
      "  9.31430000e+01 5.50783087e-02]\n",
      " [2.71141573e-01 2.42934589e-01 8.59410985e-01 ... 4.06612047e-02\n",
      "  1.33183000e+02 6.17735271e-01]\n",
      " [8.07353676e-01 1.52111783e-01 7.76784535e-01 ... 5.80036995e-02\n",
      "  1.72440000e+02 1.44911179e-01]] [[7.02435342e-01 4.14409505e-01 2.17909102e-01 ... 3.34700818e-02\n",
      "  1.18328000e+02 1.06267887e-01]\n",
      " [1.82016679e-01 5.79842601e-01 3.27383724e-01 ... 3.59045825e-02\n",
      "  1.30026000e+02 1.93946285e-01]\n",
      " [8.49479998e-01 6.05486234e-01 3.15453216e-01 ... 3.15711959e-02\n",
      "  1.25673000e+02 8.58304932e-01]\n",
      " ...\n",
      " [4.85996620e-03 6.37932987e-01 8.93932311e-01 ... 3.62283387e-02\n",
      "  9.50800000e+01 6.72629920e-01]\n",
      " [3.88495185e-02 6.04222428e-01 4.42079033e-01 ... 2.35635276e-02\n",
      "  8.09510000e+01 7.13698612e-01]\n",
      " [1.75859686e-01 5.15745249e-01 8.07032766e-01 ... 2.52133062e-02\n",
      "  1.25004000e+02 5.85561754e-01]] 992        Rock\n",
      "4537       Rock\n",
      "1731    Hip-Hop\n",
      "19         Rock\n",
      "3096       Rock\n",
      "         ...   \n",
      "1180       Rock\n",
      "3441       Rock\n",
      "1344       Rock\n",
      "4623       Rock\n",
      "1289       Rock\n",
      "Name: genre_top, Length: 3601, dtype: object 484     Rock\n",
      "4444    Rock\n",
      "1766    Rock\n",
      "4756    Rock\n",
      "4044    Rock\n",
      "        ... \n",
      "936     Rock\n",
      "3446    Rock\n",
      "4323    Rock\n",
      "4546    Rock\n",
      "3756    Rock\n",
      "Name: genre_top, Length: 1201, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Import train_test_split function and Decision tree classifier\n",
    "from sklearn.model_selection import train_test_split \n",
    "\n",
    "# Create features\n",
    "features = echo_tracks.drop([\"genre_top\", \"track_id\"], axis=1).values\n",
    "print(features, \"\\n\")\n",
    "\n",
    "# Create labels\n",
    "labels = echo_tracks[\"genre_top\"]\n",
    "print(labels, \"\\n\")\n",
    "\n",
    "# Split our data\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(features, labels, random_state=10)\n",
    "print(train_features, test_features, train_labels, test_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Normalizing the feature data\n",
    "<p>As mentioned earlier, it can be particularly useful to simplify our models and use as few features as necessary to achieve the best result. Since we didn't find any particular strong correlations between our features, we can instead use a common approach to reduce the number of features called <strong>principal component analysis (PCA)</strong>. </p>\n",
    "<p>It is possible that the variance between genres can be explained by just a few features in the dataset. PCA rotates the data along the axis of highest variance, thus allowing us to determine the relative contribution of each feature of our data towards the variance between classes. </p>\n",
    "<p>However, since PCA uses the absolute variance of a feature to rotate the data, a feature with a broader range of values will overpower and bias the algorithm relative to the other features. To avoid this, we must first normalize our train and test features. There are a few methods to do this, but a common way is through <em>standardization</em>, such that all features have a mean = 0 and standard deviation = 1 (the resultant is a z-score). </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the StandardScaler\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# Scale the features and set the values to a new variable\n",
    "scaler = StandardScaler()\n",
    "\n",
    "# Scale train_features and test_features\n",
    "scaled_train_features = scaler.fit_transform(train_features)\n",
    "scaled_test_features = scaler.transform(test_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Principal Component Analysis on our scaled data\n",
    "<p>Now that we have preprocessed our data, we are ready to use PCA to determine by how much we can reduce the dimensionality of our data. We can use <strong>scree-plots</strong> and <strong>cumulative explained ratio plots</strong> to find the number of components to use in further analyses.</p>\n",
    "<p>Scree-plots display the number of components against the variance explained by each component, sorted in descending order of variance. Scree-plots help us get a better sense of which components explain a sufficient amount of variance in our data. When using scree plots, an 'elbow' (a steep drop from one data point to the next) in the plot is typically used to decide on an appropriate cutoff.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'Principal Component #')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEKCAYAAAAcgp5RAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAATU0lEQVR4nO3df5BdZ33f8fcHOQ7BMWUGb4BaFnLA4KoYG3eR0zgF3MQeO2ai0B+DXZeWTKiqqR1CMzRRMx1KyEzHbTOZDhmDqjEmvzAqENSqWPhHA8Q0xrEkMBayLUcoot4IIjlQjIlrW/jbP85ZuKyvtGelXd/Vk/drZmfvOed5zvnea/lzzz73nOemqpAktes5ky5AkrS0DHpJapxBL0mNM+glqXEGvSQ1zqCXpMYNCvoklyfZm2Rfko1jtl+T5L7+564k549sO5Bkd5J7k+xczOIlSfPLfNfRJ1kBPARcCswAO4Crq+r+kTY/DjxQVd9IcgXw7qq6qN92AJiuqkeW5ilIko5lyBn9WmBfVe2vqieBLcC60QZVdVdVfaNfvBtYubhlSpKO1ykD2pwJPDyyPANcdIz2Pw98cmS5gNuTFPBfq2rzfAc844wzavXq1QNKkyQB7Nq165Gqmhq3bUjQZ8y6seM9SS6hC/qfGFl9cVUdTPIjwB1JHqyqO8f0XQ+sB1i1ahU7dzqcL0lDJfnK0bYNGbqZAc4aWV4JHBxzkFcDNwLrquovZ9dX1cH+9yFgK91Q0DNU1eaqmq6q6ampsW9KkqTjMCTodwDnJDk7yanAVcC20QZJVgEfB95SVQ+NrD8tyemzj4HLgC8tVvGSpPnNO3RTVUeSXAfcBqwAbqqqPUk29Ns3Ae8CXgi8LwnAkaqaBl4EbO3XnQLcXFW3LskzkSSNNe/llZMwPT1djtFL0nBJdvUn2M/gnbGS1DiDXpIaZ9BLUuMMeklqnEEvSY0bcmfsSWX1xlsmduwD1185sWNL0tF4Ri9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1LhBQZ/k8iR7k+xLsnHM9muS3Nf/3JXk/KF9JUlLa96gT7ICuAG4AlgDXJ1kzZxmfwa8vqpeDfw6sHkBfSVJS2jIGf1aYF9V7a+qJ4EtwLrRBlV1V1V9o1+8G1g5tK8kaWkNCfozgYdHlmf6dUfz88AnF9o3yfokO5PsPHz48ICyJElDDAn6jFlXYxsml9AF/a8stG9Vba6q6aqanpqaGlCWJGmIUwa0mQHOGlleCRyc2yjJq4EbgSuq6i8X0leStHSGnNHvAM5JcnaSU4GrgG2jDZKsAj4OvKWqHlpIX0nS0pr3jL6qjiS5DrgNWAHcVFV7kmzot28C3gW8EHhfEoAj/TDM2L5L9FwkSWMMGbqhqrYD2+es2zTy+G3A24b2lSQ9e7wzVpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJatyguW60OFZvvGVixz5w/ZUTO7akyfKMXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNW5Q0Ce5PMneJPuSbByz/dwkn0vyRJJ3ztl2IMnuJPcm2blYhUuShjllvgZJVgA3AJcCM8COJNuq6v6RZl8H3g787FF2c0lVPXKixUqSFm7IGf1aYF9V7a+qJ4EtwLrRBlV1qKp2AE8tQY2SpBMwJOjPBB4eWZ7p1w1VwO1JdiVZv5DiJEknbt6hGyBj1tUCjnFxVR1M8iPAHUkerKo7n3GQ7k1gPcCqVasWsHtJ0rEMOaOfAc4aWV4JHBx6gKo62P8+BGylGwoa125zVU1X1fTU1NTQ3UuS5jEk6HcA5yQ5O8mpwFXAtiE7T3JaktNnHwOXAV863mIlSQs379BNVR1Jch1wG7ACuKmq9iTZ0G/flOTFwE7g+cDTSd4BrAHOALYmmT3WzVV169I8FUnSOEPG6Kmq7cD2Oes2jTz+Gt2QzlyPAuefSIGSpBPjnbGS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDVu0HX0at/qjbdM7NgHrr9yYseW/jrwjF6SGmfQS1LjDHpJapxBL0mN88NYLXt+UCydGM/oJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjnKZYOgFOoayTgWf0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXGDgj7J5Un2JtmXZOOY7ecm+VySJ5K8cyF9JUlLa96gT7ICuAG4AlgDXJ1kzZxmXwfeDvzGcfSVJC2hIWf0a4F9VbW/qp4EtgDrRhtU1aGq2gE8tdC+kqSlNSTozwQeHlme6dcNcSJ9JUmLYEjQZ8y6Grj/wX2TrE+yM8nOw4cPD9y9JGk+Q4J+BjhrZHklcHDg/gf3rarNVTVdVdNTU1MDdy9Jms+QoN8BnJPk7CSnAlcB2wbu/0T6SpIWwbyzV1bVkSTXAbcBK4CbqmpPkg399k1JXgzsBJ4PPJ3kHcCaqnp0XN+lejKSpGcaNE1xVW0Hts9Zt2nk8dfohmUG9ZUkPXu8M1aSGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wZNgSDp5LN64y0TO/aB66+c2LH1TJ7RS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcUyBIetY5PcOzyzN6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxg0K+iSXJ9mbZF+SjWO2J8l7++33JblwZNuBJLuT3Jtk52IWL0ma37zTFCdZAdwAXArMADuSbKuq+0eaXQGc0/9cBLy//z3rkqp6ZNGqliQNNuSMfi2wr6r2V9WTwBZg3Zw264Dfrc7dwAuSvGSRa5UkHYchQX8m8PDI8ky/bmibAm5PsivJ+qMdJMn6JDuT7Dx8+PCAsiRJQwwJ+oxZVwtoc3FVXUg3vHNtkteNO0hVba6q6aqanpqaGlCWJGmIIUE/A5w1srwSODi0TVXN/j4EbKUbCpIkPUuGfGfsDuCcJGcDfw5cBfyTOW22Adcl2UL3Iew3q+qrSU4DnlNV3+ofXwa8Z/HKl6TF1eL32c4b9FV1JMl1wG3ACuCmqtqTZEO/fROwHfhpYB/wV8DP9d1fBGxNMnusm6vq1kV/FpKkoxpyRk9VbacL89F1m0YeF3DtmH77gfNPsEZJ0gnwzlhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjRsU9EkuT7I3yb4kG8dsT5L39tvvS3Lh0L6SpKU1b9AnWQHcAFwBrAGuTrJmTrMrgHP6n/XA+xfQV5K0hIac0a8F9lXV/qp6EtgCrJvTZh3wu9W5G3hBkpcM7CtJWkJDgv5M4OGR5Zl+3ZA2Q/pKkpbQKQPaZMy6GthmSN9uB8l6umEfgMeS7B1Q22I7A3jkeDvnPy5iJc9kbcfH2o7fcddnbROp7aVH2zAk6GeAs0aWVwIHB7Y5dUBfAKpqM7B5QD1LJsnOqpqeZA1HY23Hx9qO33Kuz9oWZsjQzQ7gnCRnJzkVuArYNqfNNuCf9Vff/Bjwzar66sC+kqQlNO8ZfVUdSXIdcBuwAripqvYk2dBv3wRsB34a2Af8FfBzx+q7JM9EkjTWkKEbqmo7XZiPrts08riAa4f2XcYmOnQ0D2s7PtZ2/JZzfda2AOkyWpLUKqdAkKTGGfS95TpVQ5KbkhxK8qVJ1zJXkrOSfDrJA0n2JPnFSdc0K8lzk9yT5It9bb826ZrmSrIiyReSfGLStYxKciDJ7iT3Jtk56XpGJXlBko8lebD/d/d3J10TQJJX9q/X7M+jSd4x6bpmOXTDd6dqeAi4lO5S0R3A1VV1/0QLA5K8DniM7s7jV026nlH93c8vqarPJzkd2AX87DJ53QKcVlWPJfkB4H8Dv9jfub0sJPklYBp4flW9cdL1zEpyAJiuquO+xn+pJPkd4LNVdWN/Jd/zqur/TrquUX2e/DlwUVV9ZdL1gGf0s5btVA1VdSfw9UnXMU5VfbWqPt8//hbwAMvkzud+Oo7H+sUf6H+WzVlNkpXAlcCNk67lZJHk+cDrgA8AVNWTyy3kez8JfHm5hDwY9LOcquEEJVkNvAb4k8lW8j390Mi9wCHgjqpaNrUB/wX4ZeDpSRcyRgG3J9nV37G+XPwocBj4YD/kdWOS0yZd1BhXAR+edBGjDPrO4Kka9ExJfhj4A+AdVfXopOuZVVXfqaoL6O7IXptkWQx9JXkjcKiqdk26lqO4uKoupJt19tp++HA5OAW4EHh/Vb0G+DawbD5PA+iHk34G+Oikaxll0HeGTPOgMfrx7z8APlRVH590PeP0f95/Brh8wqXMuhj4mX4sfAvw95P8/mRL+p6qOtj/PgRspRvaXA5mgJmRv8w+Rhf8y8kVwOer6i8mXcgog77jVA3Hof/A8wPAA1X1m5OuZ1SSqSQv6B//EPBTwIOTrapTVf+2qlZW1Wq6f2ufqqp/OuGyAEhyWv/BOv2wyGXAsrjiq6q+Bjyc5JX9qp8EJv7B/xxXs8yGbWDgnbGtW85TNST5MPAG4IwkM8C/r6oPTLaq77oYeAuwux8LB/jV/m7oSXsJ8Dv9FRDPAT5SVcvqMsZl6kXA1u49nFOAm6vq1smW9H1+AfhQf0K2n366leUgyfPortz7l5OuZS4vr5Skxjl0I0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINeiybJd/qZ+76U5KP95Wbj2t11nPufTvLeE6jvsaOsf3GSLUm+nOT+JNuTvOJ4j7McJHlDkh8f0O5z/e//3k9SpwYZ9FpMj1fVBf0sm08CG0Y39te0U1XzBtA4VbWzqt5+4mV+X02hu/vzM1X1sqpaA/wq3fXkJ7M3AMd8nZO8HNjXvwYv7r/nWQ0y6LVUPgu8vD+z/HSSm4Hd8L0z637bZ0bmF/9QHzokeW2Su/r55O9Jcnrf/hP99ncn+b0kn0ryp0n+Rb/+h5P8YZLP93OqzzcL6SXAU3O+GvPeqvpsOv+5/wtld5I3j9T9R0k+kuShJNcnuaavc3eSl/XtfjvJpiSf7du9sV//3CQf7Nt+Ickl/fq3Jvl4klv75/SfZmtKclmSz/XP66P9/EKzc8f/2sjzPTfdBHMbgH/d/4X190afcJIf6m9w+xTdG8IDwCv6thcs8L+zTgLeGatFl+QUujk/Zu+oXAu8qqr+bEzz1wB/m25uoT8GLk5yD/DfgDdX1Y5009M+Pqbvq4EfA04DvpDkFrqZKt9UVY8mOQO4O8m2Ovqdga+im0d/nH8AXACcD5wB7EhyZ7/tfOBv0U0hvR+4sarWpvvylV8AZr90YjXweuBlwKf7s+hrAarqvCTn0s0UOTtUdEH/mjwB7E3yW/1z/3fAT1XVt5P8CvBLwHv6Po9U1YVJ/hXwzqp6W5JNwGNV9Rtzn1RVPQ5ckOR9dFNYnEc3d/8NR3kddJLzjF6LafZMcSfwf+jnDQfuOUrIz26bqaqngXvpgvGVwFeragdAVT1aVUfG9P0fVfV4/wUZn6Z7QwnwH5LcB/wvuummj3cY5ieAD/ezYP4F8EfAa/ttO/r5+J8Avgzc3q/f3T+HWR+pqqer6k/p3hDO7ff7e/1zexD4CjAb9H9YVd+sqv9HN4/LS+nezNYAf9y/vv+8Xz9rdjK5XXOOPZ/z6OaxOY/utVejPKPXYnq8nxb4u/qRmG8fo88TI4+/Q/dvMgybJnpumwKuAaaAv1NVT6WbIfK5x9jHHuAfHWXbuOmrZ43W/fTI8tN8//9X42ocut/R1+OOqrp6nj6z7Y8pybuAf0j3V8af0M3zflmSW6vq38zXXycfz+i1HD0I/M0krwXox+fHBdi6frz7hXRjzTuAv0E31/tT/dj3S8f0G/Up4Adnx/j74702yeuBO4E3p/sCkym6bze6Z4HP5R8neU4/bv+jwN5+v9f0x3oFsKpffzR30w1pvbzv87zMf1XQt4DTx22oqvcAbwM+CFwEfLGqzjPk22XQa9npv87xzcBvJfkicAfjz8rvAW6hC8Jf7+dR/xAwne5Lra9hnqmJ+7H7NwGXpru8cg/wbrrPDLYC9wFfpHtD+OV+qtyF2Es35PNJYEM/JPM+YEWS3XSfRby1HwI6Wo2HgbcCH+6HpO6mGwI6lv8JvGnch7G919N9YL62358a5uyVOikleTdH+bBxuUjy28Anqupjk65Ff715Ri9JjfOMXpIa5xm9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJatz/B8zRAFAiIM/hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# This is just to make plots appear in the notebook\n",
    "%matplotlib inline\n",
    "\n",
    "# Import our plotting module, and PCA class\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "# Get our explained variance ratios from PCA using all features\n",
    "pca = PCA()\n",
    "pca.fit(scaled_train_features)\n",
    "exp_variance = pca.explained_variance_ratio_\n",
    "\n",
    "# plot the explained variance using a barplot\n",
    "fig, ax = plt.subplots()\n",
    "ax.bar(range(pca.n_components_), exp_variance)\n",
    "ax.set_xlabel('Principal Component #')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Further visualization of PCA\n",
    "<p>Unfortunately, there does not appear to be a clear elbow in this scree plot, which means it is not straightforward to find the number of intrinsic dimensions using this method. </p>\n",
    "<p>But all is not lost! Instead, we can also look at the <strong>cumulative explained variance plot</strong> to determine how many features are required to explain, say, about 85% of the variance (cutoffs are somewhat arbitrary here, and usually decided upon by 'rules of thumb'). Once we determine the appropriate number of components, we can perform PCA with that many components, ideally reducing the dimensionality of our data.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.lines.Line2D at 0x271e0253bb0>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhV5bn+8e9jIMwzAYEAYQggKGOY1IpKOaJVsdoBFZyqiBW1rR3sPJ5TO5ye2mpBFFRAoVonbK201QoOKCTMYQxIIIQhTAlD5jy/P7LrL4SEbDTJ2nvn/lwXV7P2Wln7Lle4XXn3u95l7o6IiES/c4IOICIitUOFLiISI1ToIiIxQoUuIhIjVOgiIjGiUVBv3LFjR09KSgrq7UVEolJaWtpBd0+oal9ghZ6UlERqampQby8iEpXMLLO6fRpyERGJESp0EZEYoUIXEYkRKnQRkRihQhcRiRE1FrqZzTWzA2a2oZr9ZmZ/MLMMM1tnZsNrP6aIiNQknCv0p4GJZ9h/JZAc+jMNmPnpY4mIyNmqsdDdfRlw+AyHTALmebkPgLZm1qW2AoqIxAJ3Z9PePB5fup33Mw7WyXvUxo1F3YDdFbazQq/trXygmU2j/CqeHj161MJbi4hErqMni3hn20GWbc1h2bYc9ucVAnDPpX24sG/HWn+/2ih0q+K1Kp+a4e6zgdkAKSkperKGiMSU0jJnbdZRlm7JYenWHNZlHaXMoU2zxlyc3JFxyQlc0i+Bc9s0rZP3r41CzwK6V9hOBLJr4bwiIhFvf14BS7eWF/i72w6Sm1+MGQxJbMt9lydzSb8EhiS2oVFc3U8qrI1CXwzMMLNFwGgg191PG24REYkFhSWlpO48wrJQiW/edwyAhFZNmDCwM+P6JXBx3460axFf79lqLHQzWwhcCnQ0syzgx0BjAHefBbwOXAVkACeB2+sqrIhIEHYePMHSrTks25rD+9sPkV9cSuM4Y2RSex66cgDj+iUw4NxWmFU1Al1/aix0d7+xhv0O3FtriUREAnaisITl2w99PJSy6/BJAHq0b84XUxK5JDmBsX060KJJYAvWVimy0oiIBKB8SuExlm3LYemWHFIzD1Nc6jRrHMeFfTpw52d6cUlyAkkdWwQd9YxU6CLSIB05UcQ7GaEphVtzOHCsfErhgHNbccdFvRjXL4ERSe1o0igu4KThU6GLSINQUlrG2qzcj8fC12YdxStOKeyXwCXJdTelsD6o0EUkZu3LLfh4Nso723LIKyjhHIMh3dty/+XJjOufwJDEtsSdE+yHmbVFhS4iMaOwpJSVHx35eCx8y/7yKYWdWjXhikHnMq5/+ZTCts3rf0phfVChi0hUyzlWyOvr97J0aw7LK00p/O7wAYzrn0D/zsFPKawPKnQRiTruzqpdR5m3fCevr99LcamT1KF8SuG4fgmM6R15UwrrQ8P7fywiUauguJTFa7J5ZvlO0rPzaNWkEVPG9OTm0T3o26lV0PECp0IXkYi369BJFnyYyfOpuzl6spj+nVvxi+vO5/PDujXIK/Hq6G9CRCJSWZmzbFsO85dn8taWA5xjxsRB5zJ1bE9G92rfIMbEz5YKXUQiSm5+MS+k7mbBB5nsPHSSji2bcN9lfblpdM+oniNeH1ToIhIRNu3NY97yTF5ZvYf84lJG9GzH1yf048rzuxDfSM+zD4cKXUQCU1xaxpL0fcx7P5MVOw/TpNE5XDe0G1PH9uT8bm2Cjhd1VOgiUu8O5BXw3IpdPPfhLg4cK6R7+2Z8/6rz+GJKYsze9FMfVOgiUi/cndTMI8xbnsnf1++lpMwZ1y+Bh2/oybh+nWLm9vsgqdBFpE7lF5Xy6po9PLM8k01782jdtBG3XpjElDE96RXhy9FGGxW6iNSJnQdPsOCD8rnjeQUlDDi3Fb+8/gImDe1K83hVT10I62/VzCYCjwBxwJPu/nCl/e2AuUAfoAC4w9031HJWEYlwZWXO0q05PLN8J0u35hBnxsTzz+XWC5NI6dlOc8frWDjPFI0DHgMmAFnASjNb7O4bKxz2PWCNu3/ezAaEjh9fF4FFJPIcPVnEC6lZzP8gk12HT5LQqgkPjE/mxlE96Nxac8frSzhX6KOADHffAWBmi4BJQMVCHwj8EsDdN5tZkpl1dvf9tR1YRCJHenYu85dn8sqaPRQUlzEqqT3fuqI/Vww6V3PHAxBOoXcDdlfYzgJGVzpmLXA98K6ZjQJ6AonAKYVuZtOAaQA9evT4hJFFJEhFJWX8fcNe5i/PJDXzCM0ax/H5Yd2YOiaJgV1bBx2vQQun0Ksa9PJK2w8Dj5jZGmA9sBooOe2b3GcDswFSUlIqn0NEItj+vAKe/bB87vjB44X07NCcH3zuPL44ojttmjcOOp4QXqFnAd0rbCcC2RUPcPc84HYAK//U46PQHxGJYu7Oio8OM295JkvS91HqzmX9O3HL2J5ckpzAOZo7HlHCKfSVQLKZ9QL2AJOBmyoeYGZtgZPuXgTcCSwLlbyIRKEThSW8smYP85dnsnnfMdo0a8wdF/diyuie9OjQPOh4Uo0aC93dS8xsBrCE8mmLc9093cymh/bPAs4D5plZKeUfln6lDjOLSB05VlDM0+/t5Ml3PyI3v5iBXVrzqxsu4Noh3WgWHxd0PKlBWPPQ3f114PVKr82q8PVyILl2o4lIfckLFfmcUJF/9rxOTB/XhxGaOx5VdLuWSAOWm/+fIt9BXkEJnz2vMw+MT+aCRK10GI1U6CINUG5+MU+99xFz3v2IYwUlTBhYXuRasja6qdBFGpDck8XMee8jnnqvvMivGNSZ+8cnM6irijwWqNBFGoDck8XMeXcHT723k2OFJUwcdC73j0/WjUAxRoUuEsOOnixizrsf8XSoyK88v7zIz+uiIo9FKnSRGHTkRKjI39/J8cISrrqgvMgHnKsij2WBFfqOnBN8+fHlp7x29eAuTB2bRH5RKbc9teK07/nCiES+mNKdwyeKuGdB2mn7p4zpyTVDupJ9NJ+v/3nNafvv+kxvPjuwM9tzjvO9l9aftv++y5O5OLkj6dm5/Oy1jaft//bE/ozo2Z60zMP8+o0tp+3/0TUDGdS1De9uO8gf39p22v7/uf4C+iS05F8b9/PEOztO2/9/Xx5K17bNeG1tNgs+yDxt/8wpI2jfIp4XUnfzl7Ss0/Y/ffsomsXHMX/5Tv66bu9p+/9891gAZi/bzpubDpyyr2njOJ65YxQAf3hzG+9lHDxlf7vm8cyaOgKAX72xmVWZR07Z36VNU34/eRgAP30tnY3Zp95X1juhBb+8fjAA331pHTtyTpyyf2DX1vz4mkEAfG3RavbmFpyyf3jPdnxn4gAAps9P48jJolP2X9S3I/ePL585e+vcFRQUl56yf/x5nZh2SR+A037uIHZ+9v6+fi8//+tG9uUVUObQvkU8gzu24cH/6q+fPWLzZ68iXaGLxIC8/BJ+9cZmnnr3IwpKymjfIp7Eds1o1lg3AzUk5h7MGlkpKSmempoayHuLxIpDxwt54p2PmLd8J/nFpVw9uCv3X96X5M6tgo4mdcTM0tw9pap9ukIXiUKHjhcy+50dzF+eSX5xKdcM7sr94/vSt5OKvCFToYtEkYPHC5m9rLzIC0tKuXZIV2ZcnkzfTi2DjiYRQIUuEgVyjhUye9l2Fnywi8KSUiYN7caMy/vSJ0FFLv+fCl0kgh04VsDspTtY8GEmRSVlXBcq8t4qcqmCCl0kAh04VsDjS3ew4INMikvLuG5YN+67PJleHVsEHU0imApdJIIcyCtg5tLtPPfhLkrKnM8P68aMy/qSpCKXMKjQRSLA/rwCZr69nYUryov8+mHlQys9O6jIJXwqdJEA7cstYNbS7Ty3YhelZc4Nw7tx72Uqcvlkwip0M5sIPEL5I+iedPeHK+1vAywAeoTO+Vt3f6qWs4rEjH25Bcx8O4OFK3dTVubcMDyRey/rq+d1yqdSY6GbWRzwGDAByAJWmtlid6+44MS9wEZ3v8bMEoAtZvZs6KHRIhKSfTSfmW9v588rd1PmzhdTEvnqpX3p3l5FLp9eOFfoo4AMd98BYGaLgEmUPwz6PxxoZeUPH2wJHAZKajmrSNQ6eLyQR/61rUKRd+erl/ZRkUutCqfQuwG7K2xnAaMrHfMosBjIBloBX3b3ssonMrNpwDSAHj16fJK8IlHnb+v28sNXN3CsoPjjIk9spyKX2hdOoVf1yO/KK3pdAawBLgf6AP80s3fc/ZR1LN19NjAbyhfnOvu4ItHj8IkifvjqBv62bi9DEtvw2y+O0aJZUqfCKfQsoHuF7UTKr8Qruh142MuXbswws4+AAcDpi/uKNABvbNjHD15ZT25+Md+6oj93X9KbRnHnBB1LYlw4hb4SSDazXsAeYDJwU6VjdgHjgXfMrDPQHzh9FX2RGHfkRBE/eS2dV9dkc3631iy4c7SeEiT1psZCd/cSM5sBLKF82uJcd083s+mh/bOAnwNPm9l6yodovuPuB6s9qUgM+ufG/Xzv5fUcPVnEgxP6Mf3SPjTWVbnUo7Dmobv768DrlV6bVeHrbOC/ajeaSHTIPVnMT19L56XVezivS2ueuX0UA7vqqlzqn+4UFfkU3tq8n+++tJ6Dx4u4f3wyMy7rS3wjXZVLMFToIp9AXkExP39tIy+kZdG/cyvm3DqS87u1CTqWNHAqdJGztHRrDg+9uI79eQXce1kf7h+fTJNGehizBE+FLhKmYwXF/M/rm1i4Yjd9O7Xk5a9exJDubYOOJfIxFbpIGN7LOMi3/7KOvbn53D2uN1//bD+aNtZVuUQWFbrIGZwoLOGXf9/Egg920btjC16YfiEjerYLOpZIlVToItVYvv0Q335xLVlH8rnz4l5884r+uiqXiKZCF6nkZFEJv35jC0+/v5OkDs15/u6xjExqH3QskRqp0EUqWLnzMN98YS2Zh05y24VJfHtif5rH65+JRAf9pIoABcWl/GbJFua+9xGJ7ZqxaNoYxvTuEHQskbOiQpcGLy3zCN96YS07Dp5g6piePHTlAFo00T8NiT76qZUGq6C4lP/751aeeGcHXdo049k7R3NR345BxxL5xFTo0iCt2X2UB59fw/acE9w4qgffu2oArZo2DjqWyKeiQpcGpbCklEf+tY1ZS7fTuXVT5t0xikv6JQQdS6RWqNClwViflcuDL6xh6/7jfCklkR9cPZDWuiqXGKJCl5hXVFLGo29t47G3t9OxZTxP3TaSywZ0CjqWSK1ToUtMS8/O5cHn17J53zGuH96NH189iDbNdVUusSmsQjezicAjlD+C7kl3f7jS/m8BN1c453lAgrsfrsWsImErLi3jT//ezh/f2ka7FvE8cUsKEwZ2DjqWSJ2qsdDNLA54DJgAZAErzWyxu2/8zzHu/hvgN6HjrwG+rjKXoGzel8eDz68lPTuPSUO78pNrBtGuRXzQsUTqXDhX6KOADHffAWBmi4BJwMZqjr8RWFg78UTCV1JaxuPLdvD7f22lddPGzJoynInndwk6lki9CafQuwG7K2xnAaOrOtDMmgMTgRnV7J8GTAPo0aPHWQUVOZNt+4/x4AtrWZeVy+cGd+Fn1w6iQ8smQccSqVfhFLpV8ZpXc+w1wHvVDbe4+2xgNkBKSkp15xAJW2mZ88Q7O/jdP7bSokkcj940jKsHdw06lkggwin0LKB7he1EILuaYyej4RapJ9tzjvPNF9ayetdRrhjUmV9cdwEJrXRVLg1XOIW+Ekg2s17AHspL+6bKB5lZG2AcMKVWE4pU4dU1e3joxfXENzqHRyYP5dohXTGr6pdJkYajxkJ39xIzmwEsoXza4lx3Tzez6aH9s0KHfh74h7ufqLO00uAVlZTx33/byDPLMxmZ1I5HbxpO59ZNg44lEhHMPZih7JSUFE9NTQ3kvSU67c3N595nV7Fq11G+cnEvHrpyAI3jzgk6lki9MrM0d0+pap/uFJWo8H7GQe5buJqC4lIeu2k4nxus6YgilanQJaK5O7OW7uA3SzbTO6Els6YMp2+nVkHHEolIKnSJWHkFxTz4/Fr+uXE/nxvchV/dMJiWepKQSLX0r0Mi0qa9edyzII2sI/n88OqB3HFRkmaxiNRAhS4R5+XVWXz3pfW0btqYhdPGMDKpfdCRRKKCCl0iRmFJKb/46ybmf5DJqF7tefSmYXRqpSmJIuFSoUtEyD6az1efXcWa3UeZdklvvnVFf01JFDlLKnQJ3HuhKYmFxaX86ebhXHWBpiSKfBIqdAlMWZkzc+l2/vcfW+iT0JJZU0fQJ6Fl0LFEopYKXQKRm18+JfFfm/ZzzZCuPHz9BbTQlESRT0X/gqTebczO455n09hzJJ8fXzOQ2y7UlESR2qBCl3r1YloW339lPW2aNWbRtDGkaEqiSK1RoUu9KCwp5WevbeTZD3cxpnd7/njjcK1dLlLLVOhS5/aEpiSu3X2Uu0NTEhtpSqJIrVOhS516Z1sO9y9cTXGp66HNInVMhS51oqzM+dPbGfzvP7eS3Kkls6aMoLemJIrUKRW61Lrck8V84/k1vLn5AJOGduWX119A83j9qInUtbAGMs1sopltMbMMM3uommMuNbM1ZpZuZktrN6ZEi/TsXK559F2Wbs3hp9cO4vdfHqoyF6knNf5LM7M44DFgApAFrDSzxe6+scIxbYE/ARPdfZeZdaqrwBK5XkjdzQ9e2UC75vH8+e6xjOjZLuhIIg1KOJdOo4AMd98BYGaLgEnAxgrH3AS85O67ANz9QG0HlchVUFzKT1/byMIVuxjbuwN/vGkYHVtqSqJIfQun0LsBuytsZwGjKx3TD2hsZm8DrYBH3H1e5ROZ2TRgGkCPHj0+SV6JMFlHTvLVZ1exLiuXey7tw4MT+mlKokhAwin0qu7J9irOMwIYDzQDlpvZB+6+9ZRvcp8NzAZISUmpfA6JMku35vDAotWUljqPTx3BFYPODTqSSIMWTqFnAd0rbCcC2VUcc9DdTwAnzGwZMATYisScsjLnj29l8Ps3t9K/cytmThlBr44tgo4l0uCF87vxSiDZzHqZWTwwGVhc6ZhXgc+YWSMza075kMym2o0qkeDoySK+8sxK/u9fW7luaDde+uqFKnORCFHjFbq7l5jZDGAJEAfMdfd0M5se2j/L3TeZ2RvAOqAMeNLdN9RlcKl/G/bkMn1BGvvzCvj5pEFMGdNTqySKRBBzD2YoOyUlxVNTUwN5bzl7z6/czQ9e3UCHFvE8dvNwhvfQlESRIJhZmrunVLVPd3zIGRUUl/KTxeksWrmbi/p24A+Th9FBUxJFIpIKXaq1+/BJ7nk2jQ178rj3sj58Y0J/4s7REItIpFKhS5X+veUAX1u0hjJ3nrglhQkDOwcdSURqoEKXU5SVOY+8uY0/vLWN/p1bMWvKCJI0i0UkKqjQ5WNHTxbxwKI1LN2aw/XDu/Hf111As/i4oGOJSJhU6ALA1v3HuGteKtlH8/nFdedz8+gempIoEmVU6MI/0vfx9T+voVl8IxZNG8OInnpws0g0UqE3YO7Oo2+VP1Xogm5tmH3LCLq0aRZ0LBH5hFToDdTJohK+9cI6/rZ+L9cN7crDNwymaWONl4tEMxV6A5R15CR3zUtjy748vnfVAO76TG+Nl4vEABV6A/PhjkPc8+wqikvLmHvbSC7tr4dLicQKFXoDsuCDTH6yOJ0eHZrzxC0p9EloGXQkEalFKvQGoKikjJ+8ls5zH+7isv4JPHLjMFo3bRx0LBGpZSr0GHfweCFfXbCKFTsPM31cH751hdZjEYlVKvQYlp6dy7R5aRw8Xsgjk4cyaWi3oCOJSB1Soceov67L5psvrKVd83j+Mv1CLkhsE3QkEaljKvQYU1bm/O6fW3n03xmM6NmOmVOG06lV06BjiUg9COeZopjZRDPbYmYZZvZQFfsvNbNcM1sT+vOj2o8qNTlWUMy0+Wk8+u8MvpzSnefuGq0yF2lAarxCN7M44DFgApAFrDSzxe6+sdKh77j71XWQUcKQeegEdz6Tyo6DJ/jptYO4Zaye9ynS0IQz5DIKyHD3HQBmtgiYBFQudAnIu9sOcu9zqzCD+XeM4sK+HYOOJCIBCGfIpRuwu8J2Vui1ysaa2Voz+7uZDarqRGY2zcxSzSw1JyfnE8SVitydOe9+xC1zP+Tc1k1ZfO/FKnORBiycK/Sqfm/3SturgJ7uftzMrgJeAZJP+yb32cBsgJSUlMrnkLNQWFLK91/ewF/SsrhiUGd+96WhtGiiz7hFGrJwGiAL6F5hOxHIrniAu+dV+Pp1M/uTmXV094O1E1MqOpBXwN0L0li96ygPjE/mgfHJnKObhUQavHAKfSWQbGa9gD3AZOCmigeY2bnAfnd3MxtF+VDOodoOK7B291GmzU/lWEEJM28ezpUXdAk6kohEiBoL3d1LzGwGsASIA+a6e7qZTQ/tnwV8AbjHzEqAfGCyu2tIpZa9vDqL77y4nk6tmvDiPRdyXpfWQUcSkQhiQfVuSkqKp6amBvLe0aa0zPnVG5uZvWwHY3q35083j6B9i/igY4lIAMwszd1TqtqnT9EiXO7JYu5btJplW3O4dWxPfnD1QBrHhXU/mIg0MCr0CJZx4Dh3zUsl68hJfnn9Bdw4qkfQkUQkgqnQI9Rbm/fzwMI1NGl8Ds/dNYaRSe2DjiQiEU6FHmHcnZlLt/ObJVsY1LU1j09NoVvbZkHHEpEooEKPIPlFpXznxXUsXpvNNUO68usbBtMsPi7oWCISJVToESL7aD7T5qeSnp3Htyf2555xfbS4loicFRV6BEjdeZjpC9IoLC5jzq0pXD6gc9CRRCQKqdADtmjFLn746gYS2zVn0bQR9O3UKuhIIhKlVOgBKS4t4+d/3ci85Zlc0i+BP04eRpvmjYOOJSJRTIUegMMnirj32VUs33GIaZf05jsTBxCnxbVE5FNSodezzfvyuPOZVA4cK+R3XxrC9cMTg44kIjFChV6P3tiwl288v5ZWTRvx/N1jGdq9bdCRRCSGqNDrQVmZ88ib23jkzW0M69GWx6eMoFNrPbxZRGqXCr2OnSgs4RvPr2FJ+n6+MCKRX1x3Pk0b62YhEal9KvQ6lJtfzE1PfMCmvXn88OqB3HFRkm4WEpE6o0KvI/lFpdz5zEq27j/Gk7pZSETqgQq9DhSXljHjuVWkZh7hD5OHqcxFpF6E9aQEM5toZlvMLMPMHjrDcSPNrNTMvlB7EaNLWZnz7b+s483NB/jZpPO5ZkjXoCOJSANRY6GbWRzwGHAlMBC40cwGVnPcryh/9miD5O78/G8beXn1Hh6c0I+pY3oGHUlEGpBwrtBHARnuvsPdi4BFwKQqjrsPeBE4UIv5ospj/87gqfd2cvtFScy4vG/QcUSkgQmn0LsBuytsZ4Ve+5iZdQM+D8w604nMbJqZpZpZak5OztlmjWjzP8jkt//YyvXDuvHDzw3UbBYRqXfhFHpVzeSVtn8PfMfdS890Inef7e4p7p6SkJAQbsaI99rabH706gbGD+jEr74wmHO0LouIBCCcWS5ZQPcK24lAdqVjUoBFoavSjsBVZlbi7q/USsoItnRrDt94fg0je7bnsZuH0zgurM+ZRURqXTiFvhJINrNewB5gMnBTxQPcvdd/vjazp4G/NoQyX7XrCNPnp9G3UyueuDVFd4CKSKBqLHR3LzGzGZTPXokD5rp7uplND+0/47h5rNq6/xi3P7WSTq2b8MwdI2nTTGuZi0iwwrqxyN1fB16v9FqVRe7ut336WJFt9+GTTJ3zIU0ancOCr4ymUysttCUiwdOA71nKOVbI1Dkfkl9UyryvjKJ7++ZBRxIRAXTr/1nJKyjmtqdWsC+vgGfvHM2Ac1sHHUlE5GO6Qg9TQXEpdz6TypZ9x5g1ZQQjerYPOpKIyCl0hR6GktIyZjy3mpU7D/P7Lw/l0v6dgo4kInIaXaHXoKzM+c6L6/nXpv387NpBTBrareZvEhEJgAr9DNyd/359Ey+uyuLrn+3H1LFJQUcSEamWCv0M/vT2dua8+xG3XZjE/eO12JaIRDYVejWe+3AXv1myhUlDu/Kjq7XYlohEPhV6FV5fv5fvv7KeS/sn8NsvDtFiWyISFVTolbyzLYcHFq1mRI92zLx5hBbbEpGoobaqYM3uo9w9P40+CS2Zc9tImsVrsS0RiR4q9JCMA8e47akVdGzZhHl3jNJiWyISdVToQNaRk0x5cgWN40KLbbXWYlsiEn0afKEfOl7ILXNWcKKohHl3jKJHBy22JSLRqUEX+rGCYm59agXZufnMvW0k53XRYlsiEr0abKEXFJdy17xUNu89xsybRzAySYttiUh0a5CLc5WUlnH/wtV8sKN8sa3LBmixLRGJfmFdoZvZRDPbYmYZZvZQFfsnmdk6M1tjZqlmdnHtR60d7s53X1rPPzbu5yfXDOS6YVpsS0RiQ41X6GYWBzwGTACygJVmttjdN1Y47E1gsbu7mQ0GngcG1EXgT+vhv2/mhbQsHhifzG0X9ar5G0REokQ4V+ijgAx33+HuRcAiYFLFA9z9uLt7aLMF4ESgWUu38/iyHdwytidf+2xy0HFERGpVOIXeDdhdYTsr9NopzOzzZrYZ+BtwR1UnMrNpoSGZ1JycnE+S9xNbtGIXD/99M9cO6cpPrhmkxbZEJOaEU+hVNd9pV+Du/rK7DwCuA35e1Yncfba7p7h7SkJCwtkl/RTe2LCX7728nnH9tNiWiMSucAo9C+heYTsRyK7uYHdfBvQxs46fMluteD/jIPcvXMPQ7m2ZOWU48Y0a7ExNEYlx4bTbSiDZzHqZWTwwGVhc8QAz62uhMQwzGw7EA4dqO+zZWpd1lLvmpdKrYwvm3jaS5vENcpamiDQQNTacu5eY2QxgCRAHzHX3dDObHto/C7gBuMXMioF84MsVPiQNRMaB49z21Erat4xn3ldG0bZ5fJBxRETqnAXVuykpKZ6amlon595zNJ8vzHyf4lLnL9PHktSxRZ28j4hIfTOzNHdPqWpfzA0oHzpeyNQ5H3K8sHyxLZW5iDQUMVXoxwtLuP3plew5ks+cW0cysKsW2xKRhiNmPiUsLCll2rxU0rPzmD11BKN6abEtEWlYYu3z0sgAAATOSURBVOIKvbTMeWDhGt7ffojffGEw48/rHHQkEZF6F/WF7u58/+X1vJG+jx9dPZDrhycGHUlEJBBRX+i/XrKFRSt3c9/lfbnjYi22JSINV1QX+uxl25n59nZuHt2Db0zoF3QcEZFARW2hP5+6m/95fTNXD+7Czyadr8W2RKTBi8pCX5K+j4deXMdnkjvyuy8NJU6LbYmIRF+hL99+iPsWrmZI97Y8PnWEFtsSEQmJujbs0DKe0b3a85QW2xIROUXUNWK/zq2Y/5XRQccQEYk4UXeFLiIiVVOhi4jECBW6iEiMUKGLiMQIFbqISIxQoYuIxAgVuohIjFChi4jEiMAeEm1mOUDmJ/z2jsDBWoxT16IpbzRlhejKG01ZIbryRlNW+HR5e7p7QlU7Aiv0T8PMUqt76nUkiqa80ZQVoitvNGWF6MobTVmh7vJqyEVEJEao0EVEYkS0FvrsoAOcpWjKG01ZIbryRlNWiK680ZQV6ihvVI6hi4jI6aL1Cl1ERCpRoYuIxIioK3Qzm2hmW8wsw8weCjrPmZjZXDM7YGYbgs5SEzPrbmb/NrNNZpZuZg8Enak6ZtbUzFaY2dpQ1p8GnSkcZhZnZqvN7K9BZzkTM9tpZuvNbI2ZpQadpyZm1tbM/mJmm0M/v2ODzlQVM+sf+jv9z588M/tarb5HNI2hm1kcsBWYAGQBK4Eb3X1joMGqYWaXAMeBee5+ftB5zsTMugBd3H2VmbUC0oDrIvHv1swMaOHux82sMfAu8IC7fxBwtDMys28AKUBrd7866DzVMbOdQIq7R8WNOmb2DPCOuz9pZvFAc3c/GnSuMwl12R5gtLt/0hssTxNtV+ijgAx33+HuRcAiYFLAmarl7suAw0HnCIe773X3VaGvjwGbgG7Bpqqalzse2mwc+hPRVyZmlgh8Dngy6CyxxMxaA5cAcwDcvSjSyzxkPLC9Nsscoq/QuwG7K2xnEaGlE83MLAkYBnwYbJLqhYYv1gAHgH+6e8RmDfk98G2gLOggYXDgH2aWZmbTgg5Tg95ADvBUaDjrSTNrEXSoMEwGFtb2SaOt0K2K1yL6yizamFlL4EXga+6eF3Se6rh7qbsPBRKBUWYWsUNaZnY1cMDd04LOEqaL3H04cCVwb2joMFI1AoYDM919GHACiPTP1uKBa4EXavvc0VboWUD3CtuJQHZAWWJOaDz6ReBZd38p6DzhCP16/TYwMeAoZ3IRcG1obHoRcLmZLQg2UvXcPTv0vweAlykf6oxUWUBWhd/Q/kJ5wUeyK4FV7r6/tk8cbYW+Ekg2s16h/8pNBhYHnCkmhD5onANscvffBZ3nTMwswczahr5uBnwW2Bxsquq5+3fdPdHdkyj/mX3L3acEHKtKZtYi9KE4oaGL/wIidpaWu+8DdptZ/9BL44GI+yC/khupg+EWKP91JWq4e4mZzQCWAHHAXHdPDzhWtcxsIXAp0NHMsoAfu/ucYFNV6yJgKrA+NDYN8D13fz3ATNXpAjwTmilwDvC8u0f0VMAo0hl4ufy/7zQCnnP3N4KNVKP7gGdDF3k7gNsDzlMtM2tO+Sy9u+vk/NE0bVFERKoXbUMuIiJSDRW6iEiMUKGLiMQIFbqISIxQoYuIxAgVuohIjFChi4jEiP8Htbu+tD7rAgUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import numpy\n",
    "import numpy as np\n",
    "\n",
    "# Calculate the cumulative explained variance\n",
    "cum_exp_variance = np.cumsum(exp_variance)\n",
    "\n",
    "# Plot the cumulative explained variance and draw a dashed line at 0.85.\n",
    "fig, ax = plt.subplots()\n",
    "ax.plot(cum_exp_variance)\n",
    "ax.axhline(y=0.85, linestyle='--')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Projecting on to our features\n",
    "<p>We saw from the plot that 6 features (remember indexing starts at 0) can explain 85% of the variance! </p>\n",
    "<p>Therefore, we can use 6 components to perform PCA and reduce the dimensionality of our train and test features.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform PCA with the chosen number of components and project data onto components\n",
    "pca = PCA(n_components = 6, random_state=10)\n",
    "\n",
    "# Fit and transform the scaled training features using pca\n",
    "train_pca = pca.fit_transform(scaled_train_features)\n",
    "\n",
    "# Fit and transform the scaled test features using pca\n",
    "test_pca = pca.transform(scaled_test_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Train a decision tree to classify genre\n",
    "<p>Now we can use the lower dimensional PCA projection of the data to classify songs into genres. </p>\n",
    "<p>Here, we will be using a simple algorithm known as a decision tree. Decision trees are rule-based classifiers that take in features and follow a 'tree structure' of binary decisions to ultimately classify a data point into one of two or more categories. In addition to being easy to both use and interpret, decision trees allow us to visualize the 'logic flowchart' that the model generates from the training data.</p>\n",
    "<p>Here is an example of a decision tree that demonstrates the process by which an input image (in this case, of a shape) might be classified based on the number of sides it has and whether it is rotated.</p>\n",
    "<p><img src=\"https://assets.datacamp.com/production/project_449/img/simple_decision_tree.png\" alt=\"Decision Tree Flow Chart Example\" width=\"350px\"></p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Decision tree classifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "\n",
    "# Train our decision tree\n",
    "tree = DecisionTreeClassifier(random_state=10)\n",
    "tree.fit(train_pca, train_labels)\n",
    "\n",
    "# Predict the labels for the test data\n",
    "pred_labels_tree = tree.predict(test_pca)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9. Compare our decision tree to a logistic regression\n",
    "<p>Although our tree's performance is decent, it's a bad idea to immediately assume that it's therefore the perfect tool for this job -- there's always the possibility of other models that will perform even better! It's always a worthwhile idea to at least test a few other algorithms and find the one that's best for our data.</p>\n",
    "<p>Sometimes simplest is best, and so we will start by applying <strong>logistic regression</strong>. Logistic regression makes use of what's called the logistic function to calculate the odds that a given data point belongs to a given class. Once we have both models, we can compare them on a few performance metrics, such as false positive and false negative rate (or how many points are inaccurately classified). </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree: \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "     Hip-Hop       0.63      0.62      0.62       235\n",
      "        Rock       0.91      0.91      0.91       966\n",
      "\n",
      "    accuracy                           0.85      1201\n",
      "   macro avg       0.77      0.77      0.77      1201\n",
      "weighted avg       0.85      0.85      0.85      1201\n",
      "\n",
      "Logistic Regression: \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "     Hip-Hop       0.77      0.54      0.64       235\n",
      "        Rock       0.90      0.96      0.93       966\n",
      "\n",
      "    accuracy                           0.88      1201\n",
      "   macro avg       0.83      0.75      0.78      1201\n",
      "weighted avg       0.87      0.88      0.87      1201\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Import LogisticRegression\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "# Train our logisitic regression\n",
    "logreg = LogisticRegression(random_state=10)\n",
    "logreg.fit(train_pca, train_labels)\n",
    "pred_labels_logit = logreg.predict(test_pca)\n",
    "\n",
    "# Create the classification report for both models\n",
    "from sklearn.metrics import classification_report\n",
    "class_rep_tree = classification_report(test_labels, pred_labels_tree)\n",
    "class_rep_log = classification_report(test_labels, pred_labels_logit)\n",
    "\n",
    "print(\"Decision Tree: \\n\", class_rep_tree)\n",
    "print(\"Logistic Regression: \\n\", class_rep_log)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 10. Balance our data for greater performance\n",
    "<p>Both our models do similarly well, boasting an average precision of 87% each. However, looking at our classification report, we can see that rock songs are fairly well classified, but hip-hop songs are disproportionately misclassified as rock songs. </p>\n",
    "<p>Why might this be the case? Well, just by looking at the number of data points we have for each class, we see that we have far more data points for the rock classification than for hip-hop, potentially skewing our model's ability to distinguish between classes. This also tells us that most of our model's accuracy is driven by its ability to classify just rock songs, which is less than ideal.</p>\n",
    "<p>To account for this, we can weight the value of a correct classification in each class inversely to the occurrence of data points for each class. Since a correct classification for \"Rock\" is not more important than a correct classification for \"Hip-Hop\" (and vice versa), we only need to account for differences in <em>sample size</em> of our data points when weighting our classes here, and not relative importance of each class. </p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Subset a balanced proportion of data points\n",
    "hop_only = echo_tracks.loc[echo_tracks['genre_top'] == 'Hip-Hop']\n",
    "rock_only = echo_tracks.loc[echo_tracks['genre_top'] == 'Rock']\n",
    "\n",
    "# subset only the rock songs, and take a sample the same size as there are hip-hop songs\n",
    "rock_only = rock_only.sample(hop_only.shape[0], random_state=10)\n",
    "\n",
    "# concatenate the dataframes hop_only and rock_only\n",
    "rock_hop_bal = pd.concat([rock_only, hop_only])\n",
    "\n",
    "# The features, labels, and pca projection are created for the balanced dataframe\n",
    "features = rock_hop_bal.drop(['genre_top', 'track_id'], axis=1) \n",
    "labels = rock_hop_bal['genre_top']\n",
    "\n",
    "# Redefine the train and test set with the pca_projection from the balanced data\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(\n",
    "    features, labels, random_state=10)\n",
    "\n",
    "train_pca = pca.fit_transform(scaler.fit_transform(train_features))\n",
    "test_pca = pca.transform(scaler.transform(test_features))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 11. Does balancing our dataset improve model bias?\n",
    "<p>We've now balanced our dataset, but in doing so, we've removed a lot of data points that might have been crucial to training our models. Let's test to see if balancing our data improves model bias towards the \"Rock\" classification while retaining overall classification performance. </p>\n",
    "<p>Note that we have already reduced the size of our dataset and will go forward without applying any dimensionality reduction. In practice, we would consider dimensionality reduction more rigorously when dealing with vastly large datasets and when computation times become prohibitively large.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree: \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "     Hip-Hop       0.82      0.77      0.79       230\n",
      "        Rock       0.78      0.82      0.80       225\n",
      "\n",
      "    accuracy                           0.80       455\n",
      "   macro avg       0.80      0.80      0.80       455\n",
      "weighted avg       0.80      0.80      0.80       455\n",
      "\n",
      "Logistic Regression: \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "     Hip-Hop       0.84      0.80      0.82       230\n",
      "        Rock       0.81      0.85      0.83       225\n",
      "\n",
      "    accuracy                           0.82       455\n",
      "   macro avg       0.82      0.82      0.82       455\n",
      "weighted avg       0.83      0.82      0.82       455\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Train our decision tree on the balanced data\n",
    "tree = DecisionTreeClassifier(random_state=10)\n",
    "tree.fit(train_pca, train_labels)\n",
    "pred_labels_tree = tree.predict(test_pca)\n",
    "\n",
    "# Train our logistic regression on the balanced data\n",
    "logreg = LogisticRegression(random_state=10)\n",
    "logreg.fit(train_pca, train_labels)\n",
    "pred_labels_logit = logreg.predict(test_pca)\n",
    "\n",
    "# compare the models\n",
    "print(\"Decision Tree: \\n\", classification_report(test_labels, pred_labels_tree))\n",
    "print(\"Logistic Regression: \\n\", classification_report(test_labels, pred_labels_logit))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 12. Using cross-validation to evaluate our models\n",
    "<p>Success! Balancing our data has removed bias towards the more prevalent class. To get a good sense of how well our models are actually performing, we can apply what's called <strong>cross-validation</strong> (CV). This step allows us to compare models in a more rigorous fashion.</p>\n",
    "<p>Before we can perform cross-validation we will need to create pipelines to scale our data, perform PCA, and instantiate our model of choice - <code>DecisionTreeClassifier</code> or <code>LogisticRegression</code>.</p>\n",
    "<p>Since the way our data is split into train and test sets can impact model performance, CV attempts to split the data multiple ways and test the model on each of the splits. Although there are many different CV methods, all with their own advantages and disadvantages, we will use what's known as <strong>K-fold</strong> CV here. K-fold first splits the data into K different, equally sized subsets. Then, it iteratively uses each subset as a test set while using the remainder of the data as train sets. Finally, we can then aggregate the results from each fold for a final model performance score.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision Tree: 0.7582417582417582 Logistic Regression: 0.782967032967033\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import KFold, cross_val_score\n",
    "from sklearn.pipeline import Pipeline\n",
    "tree_pipe = Pipeline([(\"scaler\", StandardScaler()), (\"pca\", PCA(n_components=6)), \n",
    "                      (\"tree\", DecisionTreeClassifier(random_state=10))])\n",
    "logreg_pipe = Pipeline([(\"scaler\", StandardScaler()), (\"pca\", PCA(n_components=6)), \n",
    "                        (\"logreg\", LogisticRegression(random_state=10))])\n",
    "\n",
    "# Set up our K-fold cross-validation\n",
    "kf = KFold(10)\n",
    "\n",
    "# Train our models using KFold cv\n",
    "tree_score = cross_val_score(tree_pipe, features, labels, cv=kf)\n",
    "logit_score = cross_val_score(logreg_pipe, features, labels, cv=kf)\n",
    "\n",
    "# Print the mean of each array o scores\n",
    "print(\"Decision Tree:\", np.mean(tree_score), \"Logistic Regression:\", np.mean(logit_score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}