FML1w6_feature_selection_extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature selection and feature extraction\n",
    "#### Part of the course on \"Foundations of machine learning\", Department of Mathematics and Statistics, University of Turku, Finland\n",
    "#### Lectures available on YouTube: https://youtube.com/playlist?list=PLbkSohdmxoVAZ9DEHEWHjeGK7Ei-DjKHI&si=Msu74_I0qhLrRWcu\n",
    "#### Code available on GitHub: https://github.com/ionpetre/FoundML_course_assignments\n",
    "\n",
    "#### This notebook is based on the following sources: \n",
    "> https://www.kaggle.com/code/halflingwizard/feature-selection-from-600-to-17-features#5--Recursive-feature-elimination (MATT NAMVARPOUR)\n",
    "\n",
    "> https://www.kaggle.com/code/parulpandey/penguin-dataset-the-new-iris (PARUL PANDEY)\n",
    "\n",
    "> https://www.kaggle.com/code/pascaldupont/case-study-project-r-programming-language (PSCAL DUPONT)\n",
    "\n",
    "> https://www.kaggle.com/code/arthurtok/interactive-intro-to-dimensionality-reduction (ANISOTROPIC)\n",
    "\n",
    "> https://www.kaggle.com/code/jyotiprasadpal/dimension-reduction-methods (JYOTI PRASAD PAL)\n",
    "\n",
    "> https://www.kaggle.com/code/ohseokkim/the-curse-of-dimensionality-dimension-reduction (OH SEOK KIM)\n",
    "\n",
    "Feature selection and feature extraction are fundamental steps in machine learning, very important in model development and performance. Feature selection involves identifying and choosing the most relevant features from the dataset, aiming to reduce complexity, enhance model efficiency, and avoid overfitting. By selecting the right subset of features, the model also becomes more interpretable and computationally efficient, making it easier to comprehend and implement in real-world applications. On the other hand, feature extraction involves transforming the original data into a new set of features, often with reduced dimensionality while retaining essential information. This process enhances model performance by representing the data in a more meaningful and compact manner, facilitating better generalization and interpretation. Both feature selection and feature extraction are crucial strategies to streamline model complexity, improve predictive accuracy, and facilitate efficient decision-making in machine learning.\n",
    "\n",
    "We demonstrate in this notebook the following methods:\n",
    "\n",
    "I. Feature selection\n",
    " 1. Percent Missing Values\n",
    " 2. Ammount of Variation\n",
    " 3. Pairwise Correlation\n",
    " 4. Correlation with Target\n",
    " 5. Recursive feature elimination\n",
    " 6. Select from model\n",
    "\n",
    "II. Feature extraction\n",
    " 1. PCA\n",
    " 2. LDA\n",
    " 3. t-SNE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:11.060373Z",
     "iopub.status.busy": "2021-09-09T17:34:11.059927Z",
     "iopub.status.idle": "2021-09-09T17:34:12.616363Z",
     "shell.execute_reply": "2021-09-09T17:34:12.615487Z",
     "shell.execute_reply.started": "2021-09-09T17:34:11.06034Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "import datetime\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.feature_selection import VarianceThreshold\n",
    "from sklearn.impute import KNNImputer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import f1_score, roc_auc_score, make_scorer, RocCurveDisplay\n",
    "from sklearn.model_selection import train_test_split, StratifiedKFold\n",
    "from sklearn.preprocessing import StandardScaler, Normalizer\n",
    "\n",
    "from yellowbrick.model_selection import RFECV\n",
    "from yellowbrick.classifier import ClassPredictionError, ConfusionMatrix\n",
    "\n",
    "# Import the 3 dimensionality reduction methods\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the seed of the random number generator, for reproducibility purposes\n",
    "\n",
    "import os\n",
    "\n",
    "def reset_seed(SEED = 0):\n",
    "    \"\"\"Reset the seed for every random library in use (System, numpy)\"\"\"\n",
    "\n",
    "    os.environ['PYTHONHASHSEED']=str(SEED)\n",
    "    np.random.seed(SEED)\n",
    "\n",
    "\n",
    "reset_seed(150)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the dataset: we use the UCI SECOM dataset (https://archive.ics.uci.edu/dataset/179/secom)\n",
    "> \"A complex modern semi-conductor manufacturing process is normally under consistent surveillance via the monitoring of signals/variables collected from sensors and or process measurement points. However, not all of these signals are equally valuable in a specific monitoring system. The measured signals contain a combination of useful information, irrelevant information as well as noise. It is often the case that useful information is buried in the latter two. Engineers typically have a much larger number of signals than are actually required. If we consider each type of signal as a feature, then feature selection may be applied to identify the most relevant signals. The Process Engineers may then use these signals to determine key factors contributing to yield excursions downstream in the process. This will enable an increase in process throughput, decreased time to learning and reduce the per unit production costs.\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:12.618164Z",
     "iopub.status.busy": "2021-09-09T17:34:12.617702Z",
     "iopub.status.idle": "2021-09-09T17:34:12.96194Z",
     "shell.execute_reply": "2021-09-09T17:34:12.961061Z",
     "shell.execute_reply.started": "2021-09-09T17:34:12.618132Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_openml\n",
    "\n",
    "secom_X, _ = fetch_openml(\n",
    "    data_id=43587,\n",
    "    as_frame=True,\n",
    "    return_X_y=True,\n",
    "    parser = 'auto'\n",
    ")\n",
    "\n",
    "print(f'There are {secom_X.shape[0]} records with {secom_X.shape[1]} features!')\n",
    "n_features0 = secom_X.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Separate the data into features and target. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:12.964023Z",
     "iopub.status.busy": "2021-09-09T17:34:12.963592Z",
     "iopub.status.idle": "2021-09-09T17:34:12.987282Z",
     "shell.execute_reply": "2021-09-09T17:34:12.986273Z",
     "shell.execute_reply.started": "2021-09-09T17:34:12.963994Z"
    }
   },
   "outputs": [],
   "source": [
    "secom_y = secom_X[['Pass/Fail']]\n",
    "secom_X = secom_X.drop(['Pass/Fail'], axis=1)\n",
    "\n",
    "print('The UCI SECOM dataset:')\n",
    "print(secom_X.info())\n",
    "\n",
    "print(secom_X.head())\n",
    "\n",
    "print('\\n The labels:')\n",
    "print(secom_y.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check how imbalanced the dataset is \n",
    "\n",
    "secom_y.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: The data is highly imbalanced, so we make the split train/validation/test in a stratified manner. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:13.152404Z",
     "iopub.status.busy": "2021-09-09T17:34:13.151991Z",
     "iopub.status.idle": "2021-09-09T17:34:13.183633Z",
     "shell.execute_reply": "2021-09-09T17:34:13.182484Z",
     "shell.execute_reply.started": "2021-09-09T17:34:13.152375Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train_valid, X_test, y_train_valid, y_test = train_test_split(\n",
    "    secom_X, \n",
    "    secom_y, \n",
    "    test_size=0.2, \n",
    "    random_state=2023, \n",
    "    stratify=secom_y,\n",
    "    shuffle=True\n",
    ")\n",
    "\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_valid, \n",
    "    y_train_valid, \n",
    "    test_size=0.25, \n",
    "    random_state=2023, \n",
    "    stratify=y_train_valid,\n",
    "    shuffle=True\n",
    ")\n",
    "\n",
    "del X_train_valid\n",
    "del y_train_valid\n",
    "\n",
    "# convert to pandas dataframe\n",
    "X_train = pd.DataFrame(X_train, columns=secom_X.columns)\n",
    "X_valid = pd.DataFrame(X_valid, columns=secom_X.columns)\n",
    "X_test = pd.DataFrame(X_test, columns=secom_X.columns)\n",
    "y_train = pd.DataFrame(y_train, columns=secom_y.columns)\n",
    "y_valid = pd.DataFrame(y_valid, columns=secom_y.columns)\n",
    "y_test = pd.DataFrame(y_test, columns=secom_y.columns)\n",
    "\n",
    "del secom_X\n",
    "del secom_y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: We have 939 samples in training, 314 in each of validation and test, all with 592 features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Re-labeling the Target values\n",
    "The labels are encoded in a slightly unusual way:\n",
    "> -1 corresponds to a pass and 1 corresponds to a fail and the data time stamp is for that specific test point.\n",
    "\n",
    "We change them to a more standard way so that each failure is is encoded as 0 while 1 corresponds to a pass (this of course is not crucial, it just avoid possible confusion later on). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:13.194489Z",
     "iopub.status.busy": "2021-09-09T17:34:13.193922Z",
     "iopub.status.idle": "2021-09-09T17:34:13.209419Z",
     "shell.execute_reply": "2021-09-09T17:34:13.208577Z",
     "shell.execute_reply.started": "2021-09-09T17:34:13.194451Z"
    }
   },
   "outputs": [],
   "source": [
    "y_train = y_train.replace(to_replace=[-1, 1], value=[1, 0])\n",
    "y_valid = y_valid.replace(to_replace=[-1, 1], value=[1, 0])\n",
    "y_test = y_test.replace(to_replace=[-1, 1], value=[1, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:13.211261Z",
     "iopub.status.busy": "2021-09-09T17:34:13.210705Z",
     "iopub.status.idle": "2021-09-09T17:34:13.239644Z",
     "shell.execute_reply": "2021-09-09T17:34:13.238606Z",
     "shell.execute_reply.started": "2021-09-09T17:34:13.21122Z"
    }
   },
   "outputs": [],
   "source": [
    "type_dct = {str(k): len(list(v)) for k, v in X_train.groupby(X_train.dtypes, axis=1)}\n",
    "type_dct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the column which has `object` type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:13.242724Z",
     "iopub.status.busy": "2021-09-09T17:34:13.242386Z",
     "iopub.status.idle": "2021-09-09T17:34:13.264919Z",
     "shell.execute_reply": "2021-09-09T17:34:13.263608Z",
     "shell.execute_reply.started": "2021-09-09T17:34:13.242694Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train.select_dtypes(include=['object'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity, let's drop the 'Time' feature, it will not be important for the questions asked in this notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:13.267433Z",
     "iopub.status.busy": "2021-09-09T17:34:13.267049Z",
     "iopub.status.idle": "2021-09-09T17:34:13.278906Z",
     "shell.execute_reply": "2021-09-09T17:34:13.277131Z",
     "shell.execute_reply.started": "2021-09-09T17:34:13.267401Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train_notime = X_train.drop(['Time'], axis=1)\n",
    "X_valid_notime = X_valid.drop(['Time'], axis=1)\n",
    "X_test_notime = X_test.drop(['Time'], axis=1)\n",
    "\n",
    "del X_train\n",
    "del X_valid\n",
    "del X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Count the missing values on each feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:13.281536Z",
     "iopub.status.busy": "2021-09-09T17:34:13.280903Z",
     "iopub.status.idle": "2021-09-09T17:34:13.299959Z",
     "shell.execute_reply": "2021-09-09T17:34:13.299095Z",
     "shell.execute_reply.started": "2021-09-09T17:34:13.281485Z"
    }
   },
   "outputs": [],
   "source": [
    "np.count_nonzero(X_train_notime.isna().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, 478/590 features contain missing values. Let's see the highest ammounts of missing values in a column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A nice visualization of the missing values in a dataframe\n",
    "# credit: https://www.kaggle.com/willkoehrsen/start-here-a-gentle-introduction. \n",
    "\n",
    "def missing_values_table(df):\n",
    "        # Total missing values\n",
    "        mis_val = df.isnull().sum()\n",
    "        \n",
    "        # Percentage of missing values\n",
    "        mis_val_percent = 100 * df.isnull().sum() / len(df)\n",
    "        \n",
    "        # Make a table with the results\n",
    "        mis_val_table = pd.concat([mis_val, mis_val_percent], axis=1)\n",
    "        \n",
    "        # Rename the columns\n",
    "        mis_val_table_ren_columns = mis_val_table.rename(\n",
    "        columns = {0 : 'Missing Values', 1 : '% of Total Values'})\n",
    "        \n",
    "        # Sort the table by percentage of missing descending\n",
    "        mis_val_table_ren_columns = mis_val_table_ren_columns[\n",
    "            mis_val_table_ren_columns.iloc[:,1] != 0].sort_values(\n",
    "        '% of Total Values', ascending=False).round(1)\n",
    "        \n",
    "        # Print some summary information\n",
    "        print (\"Your selected dataframe has \" + str(df.shape[1]) + \" columns.\\n\"      \n",
    "            \"There are \" + str(mis_val_table_ren_columns.shape[0]) +\n",
    "              \" columns that have missing values.\")\n",
    "        \n",
    "        # Return the dataframe with missing information\n",
    "        return mis_val_table_ren_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the missing values in the dataset\n",
    "\n",
    "X_train_missing= missing_values_table(X_train_notime)\n",
    "print(X_train_missing.to_string())\n",
    "del X_train_missing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lots of missing values. We will have to deal with them in a similar way as done in the previous week's assignment https://github.com/ionpetre/FoundML_course_assignments/blob/main/FML1w5_incomplete_data.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation model and metrics\n",
    "Throughout this notebook we will deal with dimensionality reduction, i.e., with reducing the number of features in the dataset, using various methods. This will obviously lead to some loss of information/quality and it is important to quantify this loss, so we can judge when a dimensionality reduction step is worth taking. \n",
    "\n",
    "After each dimensionality reduction step we take we will train a classifier on the transformed dataset and we will measure its performance. The accuracy sore is not appropriate for imbalanced datasets. Instead we will use the F1 score, which is the harmonic mean between the precision and the recall. It takes values between 0 and 1, with 1 being a perfect fit, and 0 showing that either the precision or the recall are 0, so the model is quite poor in at least one of the classes. We also display the ROC-AUC curve (the further it is from the main diagonal towards the top-left corner, the better the model), and the confusion matrix (showing the number of points in each class that is missclassified). Finally, we also record the training time for each version of the dataset. \n",
    "\n",
    "We imputate the missing values separately before each such evaluation. This is because the imputation may be done differently, depending on the features still retained in the dataset (albeit this is not the case with the most frequent-based technique we use in this notebook).\n",
    "\n",
    "For the classifier we will use a logistic regression (any other classifier would do). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:13.316999Z",
     "iopub.status.busy": "2021-09-09T17:34:13.316696Z",
     "iopub.status.idle": "2021-09-09T17:34:13.327542Z",
     "shell.execute_reply": "2021-09-09T17:34:13.326751Z",
     "shell.execute_reply.started": "2021-09-09T17:34:13.31697Z"
    }
   },
   "outputs": [],
   "source": [
    "def evaluate(train_df, test_df, train_target, test_target):\n",
    "    \n",
    "    # scale the data so logistic regression works better\n",
    "    scaler = StandardScaler()\n",
    "    scaler.fit(train_df)\n",
    "    train_std = pd.DataFrame(scaler.transform(train_df), columns=train_df.columns)\n",
    "    test_std = pd.DataFrame(scaler.transform(test_df), columns=test_df.columns)\n",
    "    \n",
    "    # training the model\n",
    "    logreg = LogisticRegression(\n",
    "        random_state = 175, \n",
    "        class_weight='balanced', \n",
    "        C=200, \n",
    "        dual=False, \n",
    "        solver='liblinear'\n",
    "    )\n",
    "    start_time = datetime.datetime.now()\n",
    "    logreg.fit(train_std, train_target.values.ravel())\n",
    "    elapsed = datetime.datetime.now() - start_time\n",
    "    time = int(elapsed.total_seconds()*1000)\n",
    "    \n",
    "    logreg.class_counts_ = train_target.nunique()\n",
    "    \n",
    "    # evaluation and scoring\n",
    "    y_pred = logreg.predict(test_std)\n",
    "    y_true = test_target.values.ravel()\n",
    "    f1score = f1_score(y_true, y_pred, average='micro')\n",
    "    roc_aucscore = roc_auc_score(y_true, y_pred, average='micro')\n",
    "    \n",
    "    # visualizations\n",
    "    cre = ClassPredictionError(\n",
    "        logreg, \n",
    "        isfitted=True,\n",
    "        classes=['fail', 'pass'],\n",
    "        label_encoder={0: 'fail', 1: 'pass'},\n",
    "        size=(400, 400)\n",
    "    )\n",
    "    cre.score(test_std, y_true)\n",
    "    cre.show()\n",
    "    cm = ConfusionMatrix(\n",
    "        logreg, \n",
    "        isfitted=True,\n",
    "        classes=['fail', 'pass'],\n",
    "        label_encoder={0: 'fail', 1: 'pass'},\n",
    "        percent= True,\n",
    "        size=(300, 300)\n",
    "    )\n",
    "    \n",
    "    cm.score(test_std, y_true)\n",
    "    cm.show()\n",
    "    #rocauc = plot_roc_curve(logreg, test_std, y_true)\n",
    "    RocCurveDisplay.from_estimator(logreg, test_std, y_true)\n",
    "    plt.show()\n",
    "    \n",
    "    return time, f1score, roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We check our baseline: how is the full dataset learned by our model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lists to record time and scores\n",
    "\n",
    "f1scores = []\n",
    "roc_aucscores = []\n",
    "times = []\n",
    "n_features = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:13.329147Z",
     "iopub.status.busy": "2021-09-09T17:34:13.328655Z",
     "iopub.status.idle": "2021-09-09T17:34:22.446507Z",
     "shell.execute_reply": "2021-09-09T17:34:22.445337Z",
     "shell.execute_reply.started": "2021-09-09T17:34:13.329068Z"
    }
   },
   "outputs": [],
   "source": [
    "# Replace N/A with the most frequent value in that feature\n",
    "\n",
    "from sklearn.impute import SimpleImputer\n",
    "imputer = SimpleImputer(strategy='most_frequent')\n",
    "imputer.fit(X_train_notime)\n",
    "X_imputed_train = pd.DataFrame(imputer.transform(X_train_notime), columns = X_train_notime.columns)\n",
    "X_imputed_valid = pd.DataFrame(imputer.transform(X_valid_notime), columns = X_valid_notime.columns)\n",
    "\n",
    "# lists to record time and scores\n",
    "\n",
    "time, f1score, roc_aucscore = evaluate(train_df = X_imputed_train, \n",
    "                                       test_df = X_imputed_valid, \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "n_features.append(len(X_imputed_train.columns))\n",
    "\n",
    "del X_imputed_train\n",
    "del X_imputed_valid\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though the F1 score appears quite high, we can see that the model struggles to learn the data from the ROC-AUC cruves (the score is much closer to 0.5 than to 1) and from the confusion matrix (typical situation for imbalanced data: the model learns the larger class much better than the smaller class). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I. Feature Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.1. Percent Missing Values\n",
    "We remove the columns that have most of their data missing. Imputing the missing values is not a good option because there is too little data to base the imputation on. We remove the columns missing more than 50% of their data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:22.448145Z",
     "iopub.status.busy": "2021-09-09T17:34:22.447673Z",
     "iopub.status.idle": "2021-09-09T17:34:22.453114Z",
     "shell.execute_reply": "2021-09-09T17:34:22.452208Z",
     "shell.execute_reply.started": "2021-09-09T17:34:22.448089Z"
    }
   },
   "outputs": [],
   "source": [
    "def percentna(dataframe, threshold):\n",
    "    columns = dataframe.columns[(dataframe.isna().sum()/dataframe.shape[1])>threshold]\n",
    "    return columns.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now I'm going to use this function to discover columns with more than 50% of their data missing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:22.454932Z",
     "iopub.status.busy": "2021-09-09T17:34:22.454413Z",
     "iopub.status.idle": "2021-09-09T17:34:22.497035Z",
     "shell.execute_reply": "2021-09-09T17:34:22.49596Z",
     "shell.execute_reply.started": "2021-09-09T17:34:22.454886Z"
    }
   },
   "outputs": [],
   "source": [
    "sparse_columns = percentna(X_train_notime, 0.5)\n",
    "X_train_nosparse = X_train_notime.drop(sparse_columns, axis=1)\n",
    "X_valid_nosparse = X_valid_notime.drop(sparse_columns, axis=1)\n",
    "\n",
    "n_features1 = X_train_nosparse.shape[1]\n",
    "print(f'From {len(X_train_notime.columns)} features, we are left with {n_features1} features')\n",
    "n_features.append(n_features1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the effect of removing the sparse features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace N/A with the most frequent value in that feature\n",
    "\n",
    "from sklearn.impute import SimpleImputer\n",
    "imputer = SimpleImputer(strategy='most_frequent')\n",
    "imputer.fit(X_train_nosparse)\n",
    "\n",
    "X_imputed_train = pd.DataFrame(imputer.transform(X_train_nosparse), columns = X_train_nosparse.columns)\n",
    "X_imputed_valid = pd.DataFrame(imputer.transform(X_valid_nosparse), columns = X_valid_nosparse.columns)\n",
    "\n",
    "time, f1score, roc_aucscore = evaluate(train_df = X_imputed_train, \n",
    "                                       test_df = X_imputed_valid, \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "\n",
    "del X_imputed_train\n",
    "del X_imputed_valid\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The situation is roughly the same. The model trained a little faster. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.2. Ammount of variation\n",
    "We remove the features with very low variance, i.e., those that are almost constant in every row. In fact, to be conservative, we only eliminate those with variance 0, i.e., the constant features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:37.496255Z",
     "iopub.status.busy": "2021-09-09T17:34:37.495829Z",
     "iopub.status.idle": "2021-09-09T17:34:37.525194Z",
     "shell.execute_reply": "2021-09-09T17:34:37.523937Z",
     "shell.execute_reply.started": "2021-09-09T17:34:37.496213Z"
    }
   },
   "outputs": [],
   "source": [
    "# Note: this methods does accept N/A values, no need to use imputation before applying it. \n",
    "\n",
    "selector = VarianceThreshold(threshold = 0.0)\n",
    "selector.fit(X_train_nosparse)\n",
    "\n",
    "mask = selector.get_support()\n",
    "columns = X_train_nosparse.columns\n",
    "selected_cols = columns[mask]\n",
    "n_features2 = len(selected_cols)\n",
    "print(f'From {len(columns)} features, we are left with {n_features2} features')\n",
    "n_features.append(n_features2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We remove these columns from the train/valid/test datasets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:37.535211Z",
     "iopub.status.busy": "2021-09-09T17:34:37.534815Z",
     "iopub.status.idle": "2021-09-09T17:34:37.567998Z",
     "shell.execute_reply": "2021-09-09T17:34:37.567028Z",
     "shell.execute_reply.started": "2021-09-09T17:34:37.535153Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train_var = pd.DataFrame(selector.transform(X_train_nosparse), columns = selected_cols)\n",
    "X_valid_var = pd.DataFrame(selector.transform(X_valid_nosparse), columns = selected_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the effect of removing the constant features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:37.570138Z",
     "iopub.status.busy": "2021-09-09T17:34:37.56929Z",
     "iopub.status.idle": "2021-09-09T17:34:39.179963Z",
     "shell.execute_reply": "2021-09-09T17:34:39.17919Z",
     "shell.execute_reply.started": "2021-09-09T17:34:37.570068Z"
    }
   },
   "outputs": [],
   "source": [
    "# Replace N/A with the most frequent value in that feature\n",
    "\n",
    "from sklearn.impute import SimpleImputer\n",
    "imputer = SimpleImputer(strategy='most_frequent')\n",
    "imputer.fit(X_train_var)\n",
    "\n",
    "X_imputed_train = pd.DataFrame(imputer.transform(X_train_var), columns = X_train_var.columns)\n",
    "X_imputed_valid = pd.DataFrame(imputer.transform(X_valid_var), columns = X_valid_var.columns)\n",
    "\n",
    "time, f1score, roc_aucscore = evaluate(train_df = X_imputed_train, \n",
    "                                       test_df = X_imputed_valid, \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "\n",
    "del X_imputed_train\n",
    "del X_imputed_valid\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The performance of the model is about the same, but it trained faster. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.3. Pairwise Correlation\n",
    "\n",
    "The correlation coefficient has values between -1 to 1\n",
    "- A value closer to 0 implies weaker correlation (exact 0 implying no correlation)\n",
    "- A value closer to 1 implies stronger positive correlation\n",
    "- A value closer to -1 implies stronger negative correlation\n",
    "\n",
    "If two independent features (independent = not target!) have a high absolute correlation, the information they offer for our ML model is basically the same. Correlated features in general don't improve models, so we can drop either one of them, because they are redundant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:39.18381Z",
     "iopub.status.busy": "2021-09-09T17:34:39.18335Z",
     "iopub.status.idle": "2021-09-09T17:34:39.189374Z",
     "shell.execute_reply": "2021-09-09T17:34:39.188373Z",
     "shell.execute_reply.started": "2021-09-09T17:34:39.183777Z"
    }
   },
   "outputs": [],
   "source": [
    "# We use the following function which is originally written by \n",
    "# [Krish C Naik](https://github.com/krishnaik06/Complete-Feature-Selection/blob/master/2-Feature%20Selection-%20Correlation.ipynb).\n",
    "\n",
    "# with the following function we can select highly correlated features\n",
    "# it will remove the first feature that is correlated with anything other feature\n",
    "\n",
    "def correlation(dataset, threshold):\n",
    "    col_corr = set()  # Set of all the names of correlated columns\n",
    "    corr_matrix = dataset.corr()\n",
    "    for i in range(len(corr_matrix.columns)):\n",
    "        for j in range(i):\n",
    "            if abs(corr_matrix.iloc[i, j]) > threshold: # we are interested in absolute coeff value\n",
    "                colname = corr_matrix.columns[i]  # getting the name of column\n",
    "                col_corr.add(colname)\n",
    "    return col_corr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:39.191133Z",
     "iopub.status.busy": "2021-09-09T17:34:39.190827Z",
     "iopub.status.idle": "2021-09-09T17:34:43.203672Z",
     "shell.execute_reply": "2021-09-09T17:34:43.202357Z",
     "shell.execute_reply.started": "2021-09-09T17:34:39.191093Z"
    }
   },
   "outputs": [],
   "source": [
    "corr_features = correlation(X_train_var, 0.95)\n",
    "X_train_corr = X_train_var.drop(corr_features, axis=1)\n",
    "X_valid_corr = X_valid_var.drop(corr_features, axis=1)\n",
    "n_features3 = X_train_corr.shape[1]\n",
    "\n",
    "print(f'From {len(X_train_var.columns)} features, we are left with {n_features3} features')\n",
    "n_features.append(n_features3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the effect of removing the constant features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:43.205454Z",
     "iopub.status.busy": "2021-09-09T17:34:43.205127Z",
     "iopub.status.idle": "2021-09-09T17:34:44.475881Z",
     "shell.execute_reply": "2021-09-09T17:34:44.47425Z",
     "shell.execute_reply.started": "2021-09-09T17:34:43.205425Z"
    }
   },
   "outputs": [],
   "source": [
    "# Replace N/A with the most frequent value in that feature\n",
    "\n",
    "from sklearn.impute import SimpleImputer\n",
    "imputer = SimpleImputer(strategy='most_frequent')\n",
    "imputer.fit(X_train_corr)\n",
    "\n",
    "X_imputed_train = pd.DataFrame(imputer.transform(X_train_corr), columns = X_train_corr.columns)\n",
    "X_imputed_valid = pd.DataFrame(imputer.transform(X_valid_corr), columns = X_valid_corr.columns)\n",
    "\n",
    "time, f1score, roc_aucscore = evaluate(train_df = X_imputed_train, \n",
    "                                       test_df = X_imputed_valid, \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "\n",
    "del X_imputed_train\n",
    "del X_imputed_valid\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The situation is still about the same, but the model trained even faster. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.4. Correlation with Target\n",
    "We would like our features to have high correlation with the target. if a feature has low correlation with target, it means that it is not a helpful feature for predicting the target, and hence, it should be removed. The following function will calculate correlation of each feature with the target, and then returns the columns that have a correlation less than the chosen threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:44.479583Z",
     "iopub.status.busy": "2021-09-09T17:34:44.479254Z",
     "iopub.status.idle": "2021-09-09T17:34:44.485201Z",
     "shell.execute_reply": "2021-09-09T17:34:44.48375Z",
     "shell.execute_reply.started": "2021-09-09T17:34:44.479554Z"
    }
   },
   "outputs": [],
   "source": [
    "def corrwith_target(dataframe, target, threshold):\n",
    "    cor = dataframe.corr()\n",
    "    #Correlation with output variable\n",
    "    cor_target = abs(cor[target])\n",
    "    #Selecting non correlated features\n",
    "    relevant_features = cor_target[cor_target<threshold]\n",
    "    return relevant_features.index.tolist()[:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:44.487499Z",
     "iopub.status.busy": "2021-09-09T17:34:44.486867Z",
     "iopub.status.idle": "2021-09-09T17:34:44.510317Z",
     "shell.execute_reply": "2021-09-09T17:34:44.509005Z",
     "shell.execute_reply.started": "2021-09-09T17:34:44.487452Z"
    }
   },
   "outputs": [],
   "source": [
    "# in order to find the correlation with target, we need to add target as a column to X_train_corr\n",
    "dummy_train = X_train_corr.copy(deep = True)\n",
    "dummy_train['target'] = y_train\n",
    "\n",
    "corrwith_cols = corrwith_target(dummy_train, 'target', 0.05)\n",
    "X_train_corw = X_train_corr.drop(corrwith_cols, axis=1)\n",
    "X_valid_corw = X_valid_corr.drop(corrwith_cols, axis=1)\n",
    "n_features4 = X_train_corw.shape[1]\n",
    "\n",
    "print(f'From {len(X_train_corr.columns)} features, we are left with {n_features4} features')\n",
    "n_features.append(n_features4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the effect of removing the constant features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:44.758387Z",
     "iopub.status.busy": "2021-09-09T17:34:44.757884Z",
     "iopub.status.idle": "2021-09-09T17:34:45.617005Z",
     "shell.execute_reply": "2021-09-09T17:34:45.61621Z",
     "shell.execute_reply.started": "2021-09-09T17:34:44.75834Z"
    }
   },
   "outputs": [],
   "source": [
    "# Replace N/A with the most frequent value in that feature\n",
    "\n",
    "from sklearn.impute import SimpleImputer\n",
    "imputer = SimpleImputer(strategy='most_frequent')\n",
    "imputer.fit(X_train_corw)\n",
    "\n",
    "X_imputed_train = pd.DataFrame(imputer.transform(X_train_corw), columns = X_train_corw.columns)\n",
    "X_imputed_valid = pd.DataFrame(imputer.transform(X_valid_corw), columns = X_valid_corw.columns)\n",
    "\n",
    "time, f1score, roc_aucscore = evaluate(train_df = X_imputed_train, \n",
    "                                       test_df = X_imputed_valid, \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "\n",
    "del X_imputed_train\n",
    "del X_imputed_valid\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model pays more attention to the smaller class (more of those are classified correctly), but with quite a drastic change in quality. And it trains faster still, of course. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.5. Recursive feature elimination\n",
    "We use Recursive feature elimination with cross-validation to find the optimum number of features from the remaining 38 features. RFE starts with all of the features and then eliminates features one by one (based on their importance). It stops when the desired number of features are left, or when the elimination of features no longer helps the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace N/A with the most frequent value in that feature\n",
    "\n",
    "from sklearn.impute import SimpleImputer\n",
    "imputer = SimpleImputer(strategy='most_frequent')\n",
    "imputer.fit(X_train_corw)\n",
    "\n",
    "X_imputed_train = pd.DataFrame(imputer.transform(X_train_corw), columns = X_train_corw.columns)\n",
    "X_imputed_valid = pd.DataFrame(imputer.transform(X_valid_corw), columns = X_valid_corw.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:45.618867Z",
     "iopub.status.busy": "2021-09-09T17:34:45.618377Z",
     "iopub.status.idle": "2021-09-09T17:34:45.628155Z",
     "shell.execute_reply": "2021-09-09T17:34:45.626993Z",
     "shell.execute_reply.started": "2021-09-09T17:34:45.618813Z"
    }
   },
   "outputs": [],
   "source": [
    "# Standardize each feature to have mean 0 and standard deviation 1, this will help the model fitting\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_imputed_train)\n",
    "\n",
    "X_imputed_train_std = pd.DataFrame(scaler.transform(X_imputed_train), columns=X_imputed_train.columns)\n",
    "X_imputed_valid_std = pd.DataFrame(scaler.transform(X_imputed_valid), columns=X_imputed_valid.columns)\n",
    "\n",
    "del X_imputed_train\n",
    "del X_imputed_valid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:45.649323Z",
     "iopub.status.busy": "2021-09-09T17:34:45.648835Z",
     "iopub.status.idle": "2021-09-09T17:35:26.217377Z",
     "shell.execute_reply": "2021-09-09T17:35:26.216032Z",
     "shell.execute_reply.started": "2021-09-09T17:34:45.649278Z"
    }
   },
   "outputs": [],
   "source": [
    "f1_scorer = make_scorer(\n",
    "    f1_score, \n",
    "    greater_is_better=True, \n",
    "    )\n",
    "\n",
    "\n",
    "rfecv = RFECV(\n",
    "    estimator=LogisticRegression(random_state = 175, \n",
    "                                 class_weight='balanced', \n",
    "                                 C=200, \n",
    "                                 dual=False, \n",
    "                                 solver='liblinear'\n",
    "                                ),\n",
    "    cv=StratifiedKFold(2),\n",
    "    scoring =  f1_scorer,\n",
    "    step = 1,\n",
    "    n_features_to_select = None,\n",
    "    min_features_to_select = 1,\n",
    "    n_jobs = -1,\n",
    ")\n",
    "\n",
    "rfecv.fit(X_imputed_train_std, y_train.values.ravel())\n",
    "rfecv.show() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:35:26.219071Z",
     "iopub.status.busy": "2021-09-09T17:35:26.21874Z",
     "iopub.status.idle": "2021-09-09T17:35:26.232071Z",
     "shell.execute_reply": "2021-09-09T17:35:26.230894Z",
     "shell.execute_reply.started": "2021-09-09T17:35:26.219034Z"
    }
   },
   "outputs": [],
   "source": [
    "mask = rfecv.get_support()\n",
    "columns = X_imputed_train_std.columns\n",
    "selected_cols = columns[mask]\n",
    "n_features5 = len(selected_cols)\n",
    "X_train_rfe = pd.DataFrame(rfecv.transform(X_imputed_train_std.values), columns = selected_cols)\n",
    "X_valid_rfe = pd.DataFrame(rfecv.transform(X_imputed_valid_std.values), columns = selected_cols)\n",
    "\n",
    "print(f'From {len(X_imputed_train_std.columns)} features, we are left with {n_features5} features')\n",
    "n_features.append(n_features5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the effect of the recursive feature elimination"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:35:26.233904Z",
     "iopub.status.busy": "2021-09-09T17:35:26.233589Z",
     "iopub.status.idle": "2021-09-09T17:35:26.85884Z",
     "shell.execute_reply": "2021-09-09T17:35:26.857012Z",
     "shell.execute_reply.started": "2021-09-09T17:35:26.233874Z"
    }
   },
   "outputs": [],
   "source": [
    "time, f1score, roc_aucscore = evaluate(train_df = X_train_rfe, \n",
    "                                       test_df = X_valid_rfe, \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "\n",
    "\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I.6. Select from model\n",
    "We can also select features based on the feature importance ranking reported by a classifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "\n",
    "clf = RandomForestClassifier(\n",
    "    n_estimators = 100, \n",
    "    criterion = 'entropy',\n",
    "    max_depth = 5,\n",
    "    min_samples_split = 5,\n",
    "    random_state = 180,\n",
    "    n_jobs = -1,\n",
    ")\n",
    "\n",
    "clf = clf.fit(X_train_rfe, y_train.values.ravel())\n",
    "\n",
    "model = SelectFromModel(clf, prefit=True)\n",
    "feature_idx = model.get_support()\n",
    "feature_name = X_train_rfe.columns[feature_idx]\n",
    "n_features6 = len(feature_name)\n",
    "\n",
    "X_train_RF = pd.DataFrame(model.transform(X_train_rfe.values), columns=feature_name)\n",
    "X_valid_RF = pd.DataFrame(model.transform(X_valid_rfe.values), columns=feature_name)\n",
    "\n",
    "print(f'From {len(X_imputed_train_std.columns)} features, we are left with {n_features6} features')\n",
    "n_features.append(n_features6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluate the effect of the Select from Model elimination\n",
    "\n",
    "To keep the comparison similar, we train a logistic regression model on the features selected by the RF model, rather than using the score of the RF model itself. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time, f1score, roc_aucscore = evaluate(train_df = X_train_RF, \n",
    "                                       test_df = X_valid_RF, \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results of the feature selection\n",
    "\n",
    "We went down from 589 features to just 12 features. The training time clearly goes down drastically. The information loss is also considerable, especially from step 4 onwards, as seen through the f1 score. That step obviously removes too many features, but it was good enough for the purpose of this demo. \n",
    ">For your own curiosity, modify the parameter we used in step 4 for the correlation between the features and the target, from 0,05 to 0,03 or smaller. What do you expect to see in this step and in the next?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature extraction\n",
    "\n",
    "Whereas in feature selection the focus was on deciding which features of the original data to retain, in feature extraction (or feature engineering), the focus is on identifying a smaller number of (possibly new) features that retain as much as possibly the \"quality\" of the dataset. These new features depend on the original features, but they can be very different from them. \n",
    "\n",
    "We start the demo of the feature extraction from the 'X_train_nosparse' version of the data, i.e., that where we just removed the columns missing more than 50% of the data. \n",
    "\n",
    "We imputate the missing values, standardize the data, and proceed with the demo. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:34:13.329147Z",
     "iopub.status.busy": "2021-09-09T17:34:13.328655Z",
     "iopub.status.idle": "2021-09-09T17:34:22.446507Z",
     "shell.execute_reply": "2021-09-09T17:34:22.445337Z",
     "shell.execute_reply.started": "2021-09-09T17:34:13.329068Z"
    }
   },
   "outputs": [],
   "source": [
    "# Replace N/A with the most frequent value in that feature\n",
    "\n",
    "imputer = SimpleImputer(strategy='most_frequent')\n",
    "imputer.fit(X_train_nosparse)\n",
    "\n",
    "X_imputed_train = pd.DataFrame(imputer.transform(X_train_nosparse), columns = X_train_nosparse.columns)\n",
    "X_imputed_valid = pd.DataFrame(imputer.transform(X_valid_nosparse), columns = X_valid_nosparse.columns)\n",
    "\n",
    "# Standardize each feature to have mean 0 and standard deviation 1, this will help the model fitting\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(X_imputed_train)\n",
    "\n",
    "X_train = pd.DataFrame(scaler.transform(X_imputed_train), columns=X_imputed_train.columns)\n",
    "X_valid = pd.DataFrame(scaler.transform(X_imputed_valid), columns=X_imputed_valid.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### II.1 PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply PCA\n",
    "\n",
    "pca = PCA().fit(X_train.values)\n",
    "pca_var_df = pd.DataFrame({'Information': pca.explained_variance_ratio_,\n",
    "                           'PCs': np.arange(pca.n_components_)\n",
    "                          })\n",
    "plt.figure(figsize = (25,8))\n",
    "sns.barplot(x = 'PCs',y = 'Information',data = pca_var_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the number of principal components required for an explained variance of at least 80%\n",
    "\n",
    "var = 0\n",
    "for i in range(pca.n_components_): \n",
    "    var += pca.explained_variance_ratio_[i]\n",
    "    if var > 0.8:\n",
    "        break\n",
    "        \n",
    "n_comp_80var = i+1\n",
    "print('For an 80% variance the number of PCs we need is', n_comp_80var)\n",
    "\n",
    "pca_var_df = pd.DataFrame({'Information': pca.explained_variance_ratio_[0:n_comp_80var],\n",
    "                           'PCs': np.arange(n_comp_80var)\n",
    "                          })\n",
    "plt.figure(figsize = (25,8))\n",
    "sns.barplot(x = 'PCs',y = 'Information',data = pca_var_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use PCA for visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the first 2 principal components\n",
    "X_train_2comps = PCA(n_components = 2).fit_transform(X_train)\n",
    "\n",
    "fig = plt.figure(figsize = (10,7))\n",
    "plt.plot(X_train_2comps[:,0], X_train_2comps[:,1], marker = 'o', color = 'teal', alpha = .5, linewidth = 0)\n",
    "plt.xlabel('Principal Component 1', fontsize = 14)\n",
    "plt.ylabel('Principal Component 2', fontsize = 14)\n",
    "plt.grid(linestyle = '--')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Colour code the 2 Principal Component plot depending on whether target is 1 or 0\n",
    "\n",
    "# Store binary y and X in a dataframe \n",
    "plot_2comps = pd.DataFrame(X_train_2comps, columns = ['PC1', 'PC2'])\n",
    "\n",
    "plot_2comps['Pass'] = y_train.to_numpy()\n",
    "\n",
    "X_2comps_0 = plot_2comps[plot_2comps['Pass'] == 0][['PC1', 'PC2']].copy()\n",
    "X_2comps_1 = plot_2comps[plot_2comps['Pass'] == 1][['PC1', 'PC2']].copy()\n",
    "\n",
    "fig = plt.figure(figsize = (10,7))\n",
    "plt.plot(X_2comps_0['PC1'], X_2comps_0['PC2'], marker = 'o', color = 'red', alpha = .5,\n",
    "         linewidth = 0, label = 'Pass = 0')\n",
    "plt.plot(X_2comps_1['PC1'], X_2comps_1['PC2'], marker = 'o', color = 'blue', alpha = .5,\n",
    "         linewidth = 0, label = 'Pass = 1')\n",
    "plt.xlabel('Principal Component 1', fontsize = 14)\n",
    "plt.ylabel('Principal Component 2', fontsize = 14)\n",
    "plt.legend()\n",
    "#plt.grid(linestyle = '--')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: the data is not well separated by the 2 PCs. This is likely because they retain too little of the variance of the data (recall we needed all the 81 first PCs to retain 80% of the variance). \n",
    "\n",
    "Run the cell about with only one of the classes displayed so you can realise where they are placed in this 2D plot. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use PCA for dimensionality reduction for other models to train on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Project the data onto the PCs that offer 80% variance\n",
    "\n",
    "pca80 = PCA(n_components=n_comp_80var).fit(X_train.values)\n",
    "pca80.fit(X_train.values)\n",
    "\n",
    "X_train_pca80 = pd.DataFrame(pca80.transform(X_train.values))\n",
    "X_valid_pca80 = pd.DataFrame(pca80.transform(X_valid.values))\n",
    "\n",
    "time, f1score, roc_aucscore = evaluate(train_df = X_train_pca80, \n",
    "                                       test_df = X_valid_pca80, \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "n_features.append(pca80.n_components_)\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### II.2 LDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use LDA for visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda = LDA()\n",
    "lda.fit(X_train, y_train.values.ravel())\n",
    "\n",
    "print(lda.explained_variance_ratio_)\n",
    "\n",
    "X_train_lda = lda.transform(X_train)\n",
    "X_valid_lda = lda.transform(X_valid)\n",
    "\n",
    "plot_lda = pd.DataFrame(X_train_lda, columns = ['LDA'])\n",
    "plot_lda['Pass'] = y_train.to_numpy()\n",
    "\n",
    "plot_lda_0 = plot_lda[plot_lda['Pass'] == 0][['LDA']].copy()\n",
    "plot_lda_1 = plot_lda[plot_lda['Pass'] == 1][['LDA']].copy()\n",
    "\n",
    "\n",
    "plt.figure(figsize=(10,2))\n",
    "plt.scatter( plot_lda_0, [0] * plot_lda_0.shape[0], marker = 'o', color = 'red', alpha = .5,\n",
    "         linewidth = 1, label = 'Pass = 0')\n",
    "plt.scatter( plot_lda_1, [0] * plot_lda_1.shape[0], marker = 'o', color = 'blue', alpha = .5,\n",
    "         linewidth = 1, label = 'Pass = 1')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use LDA for dimensionality reduction for other models to train on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time, f1score, roc_aucscore = evaluate(train_df = pd.DataFrame(X_train_lda), \n",
    "                                       test_df = pd.DataFrame(X_valid_lda), \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "n_features.append(lda.explained_variance_ratio_.shape[0])\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### II.3 t-SNE\n",
    "t-Distributed Stochastic Neighbor Embedding (t-SNE) is a technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets. Points that are close to the original feature space are also expressed in a two-dimensional plane after compression. Since the nonlinear relationship can be identified, the model performance can be improved by adding the compression results expressed by these t-SNEs to the original features. However, since the computation cost is high, it is not suitable for compression exceeding two or three dimensions.\n",
    "\n",
    ">Link to the author Laurens van der Maaten's page (it includes links to his original paper, to lectures and implementations): https://lvdmaaten.github.io/tsne/\n",
    "\n",
    ">Link to a simplified presentation of the main ideas in t-SNE: https://youtu.be/NEaUSP4YerM?si=239qIfc12m7SCFS4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne = TSNE(\n",
    "    n_components = 2,\n",
    "    perplexity = 5,\n",
    ")\n",
    "\n",
    "X_train_tsne = tsne.fit_transform(X_train)\n",
    "X_valid_tsne = tsne.fit_transform(X_valid)\n",
    "\n",
    "# Colour code the t-SNE plot depending on whether target is 1 or 0\n",
    "\n",
    "# Store binary y and X in a dataframe \n",
    "plot_tsne = pd.DataFrame(X_train_tsne, columns = ['tSNE1', 'tSNE2'])\n",
    "\n",
    "plot_tsne['Pass'] = y_train.to_numpy()\n",
    "\n",
    "X_train_tsne_0 = plot_tsne[plot_tsne['Pass'] == 0][['tSNE1', 'tSNE2']].copy()\n",
    "X_train_tsne_1 = plot_tsne[plot_tsne['Pass'] == 1][['tSNE1', 'tSNE2']].copy()\n",
    "\n",
    "fig = plt.figure(figsize = (10,7))\n",
    "plt.plot(X_train_tsne_0['tSNE1'], X_train_tsne_0['tSNE2'], marker = 'o', color = 'red', alpha = .5,\n",
    "         linewidth = 0, label = 'Pass = 0')\n",
    "plt.plot(X_train_tsne_1['tSNE1'], X_train_tsne_1['tSNE1'], marker = 'o', color = 'blue', alpha = .5,\n",
    "         linewidth = 0, label = 'Pass = 1')\n",
    "plt.xlabel('t-SNE Component 1', fontsize = 14)\n",
    "plt.ylabel('t-SNE Component 2', fontsize = 14)\n",
    "plt.legend()\n",
    "#plt.grid(linestyle = '--')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use t-SNE for dimensionality reduction for other models to train on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time, f1score, roc_aucscore = evaluate(train_df = pd.DataFrame(X_train_tsne), \n",
    "                                       test_df = pd.DataFrame(X_valid_tsne), \n",
    "                                       train_target=y_train, \n",
    "                                       test_target=y_valid\n",
    "                                      )\n",
    "print(f' Training time: {time}ms\\n F1 Score: {f1score}\\n ROC-AUC Score: {roc_aucscore}')\n",
    "n_features.append(tsne.embedding_.shape[1])\n",
    "f1scores.append(f1score)\n",
    "roc_aucscores.append(roc_aucscore)\n",
    "times.append(time)\n",
    "del time\n",
    "del f1score\n",
    "del roc_aucscore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of feature selection and feature engineering: how did we do?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-09-09T17:35:26.860329Z",
     "iopub.status.busy": "2021-09-09T17:35:26.860026Z",
     "iopub.status.idle": "2021-09-09T17:35:27.377879Z",
     "shell.execute_reply": "2021-09-09T17:35:27.376845Z",
     "shell.execute_reply.started": "2021-09-09T17:35:26.8603Z"
    }
   },
   "outputs": [],
   "source": [
    "xlabels = ['Full', 'NoSparse', 'NoVar', 'NoPairCorr', 'NoTarCorr', 'RFE', 'Model', 'PCA', 'LDA', 'tSNE']\n",
    "\n",
    "fig, (ax2, ax0, ax1, ax4) = plt.subplots(4, 1, figsize=(8, 8))\n",
    "fig.tight_layout()\n",
    "\n",
    "ax2.plot(n_features, label='# of features', c='g')\n",
    "ax2.set(ylabel='# of features')\n",
    "#ax2.set(xlabel='Feature selection step')\n",
    "ax2.set_xticks(np.arange(len(n_features)), labels=xlabels)\n",
    "ax2.legend()\n",
    "\n",
    "ax0.plot(times, label='Training time', c='b')\n",
    "ax0.set(ylabel='Training Time (ms)')\n",
    "ax0.set_xticks(np.arange(len(times)), labels=xlabels)\n",
    "ax0.legend()\n",
    "\n",
    "ax1.plot(f1scores, label='F1 Score', c='r')\n",
    "ax1.set(ylabel='F1 score')\n",
    "#ax1.set(xlabel='Feature selection step')\n",
    "ax1.set_xticks(np.arange(len(f1scores)), labels=xlabels)\n",
    "ax1.set_ylim([0.4, 1])\n",
    "ax1.legend()\n",
    "\n",
    "ax4.plot(roc_aucscores, label='ROC-AUC Score', c='m')\n",
    "ax4.set(ylabel='ROC-AUC score')\n",
    "#ax4.set(xlabel='Feature selection step')\n",
    "ax4.set_xticks(np.arange(len(roc_aucscores)), labels=xlabels)\n",
    "ax4.set_ylim([0.4, 1])\n",
    "ax4.legend()\n",
    "\n",
    "#fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Conclusion.** We have a clear winner on this dataset: LDA produced a single engineered feature that offers a model of the same quality as the full set of almost 600 features! Note though that the performance of the model on the minority class remained limited throughout our attempts. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bonus challenge.** Send me any model (using any architecture and any data processing) for the UCI SECOM dataset with an ROC_AUC score over 0.7 on the test dataset, for an extra 10 points under \"Assignments\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Challenge: Apply feature selection and engineering to the Penguin dataset\n",
    "The dataset only has 7 features and it can of course be used as is for machine learning. It is still worth it to do feature selection/engineering for visualization purposes: see how the data is distributed in 2 well-chosen dimensions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Meet the penguins (Artwork by @allison_horst https://github.com/allisonhorst/penguins)\n",
    "\n",
    "![](https://imgur.com/orZWHly.png)\n",
    "\n",
    "The Palmer Archipelago (Antarctica) penguin dataset is great for data exploration & visualization, as an alternative to iris. Data were collected and made available by Dr. Kristen Gorman and the Palmer Station, Antarctica LTER, a member of the Long Term Ecological Research Network. Please see https://github.com/allisonhorst/palmerpenguins for more information.\n",
    "\n",
    "The dataset consists of 7 columns. \n",
    "\n",
    "* **species**: penguin species (Chinstrap, Adélie, or Gentoo)\n",
    "* **culmen_length_mm**: culmen length (mm)\n",
    "* **culmen_depth_mm**: culmen depth (mm)\n",
    "* **flipper_length_mm**: flipper length (mm)\n",
    "* **body_mass_g**: body mass (g)\n",
    "* **island**: island name (Dream, Torgersen, or Biscoe) in the Palmer Archipelago (Antarctica)\n",
    "* **sex**: penguin sex\n",
    "\n",
    "<img src=\"https://i.imgur.com/trfGnPa.png\" width=\"800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Penguin body parts: flippers, culmen (Artwork by @allison_horst https://github.com/allisonhorst/penguins)\n",
    "\n",
    "<img src=\"https://i.imgur.com/ie2WP7w.jpg\" width=\"800px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1: data analytics\n",
    "- Load the dataset from OpenML (ID 42585). Check the X and the y parts returned by the function call. In this notebook we will consider this as a classifier with the target being the species name. We will do both visualization, as well as training classifiers. \n",
    "- Q1: Do you consider this data to be imbalanced? \n",
    "- Q2: How many features do you have (consider the target as a separate column, not part of the features)? \n",
    "- Q3: How many features contain missing values? \n",
    "- Check the 'sex' feature with penguin_X['sex'].value_counts(). It contains a datapoint with 'sex' set to a value that does not make sense. Remove that value (change it to appear as a missing value) with the call penguin_X['sex'] = penguin_X['sex'].cat.remove_categories(['WRITE_WEIRD_VALUE_HERE']) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encode the categorical features 'island' and 'sex' with numerical encodings.\n",
    "# 'sex' has N/A values, and they should remain N/A also after the change in the encoding.\n",
    "# One solution is to save the rows with N/A, transform the others, then put back the rows with N/A.\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "lb = LabelEncoder()\n",
    "\n",
    "penguin_X['island'] = lb.fit_transform(penguin_X['island'])\n",
    "\n",
    "peng_na = penguin_X[penguin_X['sex'].isna()].copy()\n",
    "penguin_X.dropna(subset='sex', axis=0, inplace=True)\n",
    "penguin_X['sex'] = lb.fit_transform(penguin_X['sex'])\n",
    "\n",
    "penguin_X = pd.concat([penguin_X, peng_na])\n",
    "del peng_na\n",
    "\n",
    "penguin_y = pd.DataFrame(lb.fit_transform(penguin_y))\n",
    "\n",
    "penguin_X.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 1: feature selection/engineering\n",
    "\n",
    "- Split your dataset into train/validation/test using the same percentages as in the SECOM dataset, using a stratified approach or not, depending on your answer to Q1. \n",
    "- Q4: Did you use a stratified split? \n",
    "- Imputate the missing values in your train/valid datasets using the most frequent-vased strategy. \n",
    "- Standardize the train/valid datasets using a Standard Scaler. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the code below to evaluate the effects of the feature selection/engineering methods. This is slightly different than the SECOM dataset, because we have a 3-class classification dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate2(train_df, test_df, train_target, test_target):\n",
    "    \n",
    "    # training the model\n",
    "    logreg = LogisticRegression(\n",
    "        random_state = 175, \n",
    "        class_weight='balanced', \n",
    "        C=200, \n",
    "        dual=False, \n",
    "        multi_class='multinomial'\n",
    "    )\n",
    "    start_time = datetime.datetime.now()\n",
    "    logreg.fit(train_df, train_target.values.ravel())\n",
    "    elapsed = datetime.datetime.now() - start_time\n",
    "    time = int(elapsed.total_seconds()*1000)\n",
    "    \n",
    "    logreg.class_counts_ = train_target.nunique()\n",
    "    \n",
    "    # evaluation and scoring\n",
    "    y_pred_prob = logreg.predict_proba(test_df)\n",
    "    y_pred = logreg.predict(test_df)\n",
    "    y_true = test_target.values.ravel()\n",
    "    f1score = f1_score(y_true, y_pred, average='micro')\n",
    "    roc_aucscore = roc_auc_score(y_true, y_pred_prob, average='micro', multi_class='ovr')\n",
    "    \n",
    "    # visualizations\n",
    "    cre = ClassPredictionError(\n",
    "        logreg, \n",
    "        isfitted=True,\n",
    "        classes=['Adelie', 'Chinstrap', 'Gentoo'],\n",
    "        label_encoder={0: 'Adelie', 1: 'Chinstrap', 2:'Gentoo'},\n",
    "        size=(400, 400)\n",
    "    )\n",
    "    cre.score(test_df, y_true)\n",
    "    cre.show()\n",
    "    cm = ConfusionMatrix(\n",
    "        logreg, \n",
    "        isfitted=True,\n",
    "        percent= True,\n",
    "        size=(300, 300)\n",
    "    )\n",
    "    \n",
    "    cm.score(test_df, y_true)\n",
    "    cm.show()\n",
    "    \n",
    "    return time, f1score, roc_aucscore\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the standardized datasets in each of the evaluation to be done. \n",
    "- Evaluate the learnability of the dataset with all its features in.\n",
    "- Do feature selection based on a random forest model (code below). \n",
    "- Q4: Which features did the random forest select? \n",
    "- Evaluate the learnability of the dataset projected on the features selected by the RF model. \n",
    "- Apply PCA with as few components you need in order to preserve at least 80% of the data variance\n",
    "- Q5: How many PCs did you need? \n",
    "- Evaluate the learnability of the dataset transformed through the PCA you just trained. \n",
    "- Visualize the dataset along the first two PCs.\n",
    "- Apply LDA and visualize the data along the 2 components you get. \n",
    "- Evaluate the learnability of the dataset transformed through the LDA you just trained. \n",
    "- Apply t-SNE with 2 components and visualize the data along the 2 components you get. \n",
    "- Evaluate the learnability of the dataset transformed through the t-SNE you just trained. \n",
    "- Plot the results of the 5 evaluations you did. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the code below for the model-based feature selection with random forests. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "\n",
    "clf = RandomForestClassifier(\n",
    "    n_estimators = 100, \n",
    "    criterion = 'entropy',\n",
    "    max_depth = 5,\n",
    "    min_samples_split = 5,\n",
    "    random_state = 180,\n",
    "    n_jobs = -1,\n",
    ")\n",
    "\n",
    "clf = clf.fit(X_train_std, y_train.values.ravel())\n",
    "\n",
    "model = SelectFromModel(clf, prefit=True)\n",
    "feature_idx = model.get_support()\n",
    "feature_name = X_train_std.columns[feature_idx]\n",
    "n_features6 = len(feature_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualization code for PCA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the first 2 principal components\n",
    "X_train_2comps = PCA(n_components = 2).fit_transform(X_train_std)\n",
    "\n",
    "# Colour code the 2 Principal Component plot\n",
    "\n",
    "# Store binary y and X in a dataframe \n",
    "plot_2comps = pd.DataFrame(X_train_2comps, columns = ['PC1', 'PC2'])\n",
    "\n",
    "plot_2comps['species'] = y_train.to_numpy()\n",
    "\n",
    "X_2comps_0 = plot_2comps[plot_2comps['species'] == 0][['PC1', 'PC2']].copy()\n",
    "X_2comps_1 = plot_2comps[plot_2comps['species'] == 1][['PC1', 'PC2']].copy()\n",
    "X_2comps_2 = plot_2comps[plot_2comps['species'] == 2][['PC1', 'PC2']].copy()\n",
    "\n",
    "fig = plt.figure(figsize = (10,10))\n",
    "plt.plot(X_2comps_0['PC1'], X_2comps_0['PC2'], marker = 'o', color = 'red', alpha = .5,\n",
    "         linewidth = 0, label = 'Adelie')\n",
    "plt.plot(X_2comps_1['PC1'], X_2comps_1['PC2'], marker = 'o', color = 'blue', alpha = .5,\n",
    "         linewidth = 0, label = 'Chinstrap')\n",
    "plt.plot(X_2comps_2['PC1'], X_2comps_2['PC2'], marker = 'o', color = 'green', alpha = .5,\n",
    "         linewidth = 0, label = 'Gentoo')\n",
    "plt.xlabel('Principal Component 1', fontsize = 14)\n",
    "plt.ylabel('Principal Component 2', fontsize = 14)\n",
    "plt.legend()\n",
    "#plt.grid(linestyle = '--')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualization code for LDA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda = LDA()\n",
    "lda.fit(X_train_std, y_train.values.ravel())\n",
    "\n",
    "print(lda.explained_variance_ratio_)\n",
    "\n",
    "X_train_lda = lda.transform(X_train_std)\n",
    "X_valid_lda = lda.transform(X_valid_std)\n",
    "\n",
    "plot_lda = pd.DataFrame(X_train_lda, columns = ['LDA1',  'LDA2'])\n",
    "plot_lda['species'] = y_train.to_numpy()\n",
    "\n",
    "plot_lda_0 = plot_lda[plot_lda['species'] == 0][['LDA1', 'LDA2']].copy()\n",
    "plot_lda_1 = plot_lda[plot_lda['species'] == 1][['LDA1', 'LDA2']].copy()\n",
    "plot_lda_2 = plot_lda[plot_lda['species'] == 2][['LDA1', 'LDA2']].copy()\n",
    "\n",
    "\n",
    "plt.figure(figsize=(10,10))\n",
    "plt.scatter( plot_lda_0['LDA1'], plot_lda_0['LDA2'], marker = 'o', color = 'red', alpha = .5,\n",
    "         linewidth = 1, label = 'Adelie')\n",
    "plt.scatter( plot_lda_1['LDA1'], plot_lda_1['LDA2'], marker = 'o', color = 'blue', alpha = .5,\n",
    "         linewidth = 1, label = 'Chinstrap')\n",
    "plt.scatter( plot_lda_2['LDA1'], plot_lda_2['LDA2'], marker = 'o', color = 'green', alpha = .5,\n",
    "         linewidth = 1, label = 'Gentoo')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualization code for t-SNE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne = TSNE(\n",
    "    n_components = 2,\n",
    "    perplexity = 5,\n",
    ")\n",
    "\n",
    "X_train_tsne = tsne.fit_transform(X_train_std)\n",
    "X_valid_tsne = tsne.fit_transform(X_valid_std)\n",
    "\n",
    "# Colour code the t-SNE plot depending on the target\n",
    "\n",
    "# Store binary y and X in a dataframe \n",
    "plot_tsne = pd.DataFrame(X_train_tsne, columns = ['tSNE1', 'tSNE2'])\n",
    "\n",
    "plot_tsne['species'] = y_train.to_numpy()\n",
    "\n",
    "X_train_tsne_0 = plot_tsne[plot_tsne['species'] == 0][['tSNE1', 'tSNE2']].copy()\n",
    "X_train_tsne_1 = plot_tsne[plot_tsne['species'] == 1][['tSNE1', 'tSNE2']].copy()\n",
    "X_train_tsne_2 = plot_tsne[plot_tsne['species'] == 2][['tSNE1', 'tSNE2']].copy()\n",
    "\n",
    "fig = plt.figure(figsize = (10,10))\n",
    "plt.plot(X_train_tsne_0['tSNE1'], \n",
    "         X_train_tsne_0['tSNE2'], \n",
    "         marker = 'o', \n",
    "         color = 'red', \n",
    "         alpha = .5,\n",
    "         linewidth = 0, \n",
    "         label = 'Adelie'\n",
    "        )\n",
    "plt.plot(X_train_tsne_1['tSNE1'], \n",
    "         X_train_tsne_1['tSNE1'], \n",
    "         marker = 'o', \n",
    "         color = 'blue', \n",
    "         alpha = .5,\n",
    "         linewidth = 0, \n",
    "         label = 'Chinstrap'\n",
    "        )\n",
    "plt.plot(X_train_tsne_2['tSNE1'], \n",
    "         X_train_tsne_2['tSNE1'], \n",
    "         marker = 'o', \n",
    "         color = 'green', \n",
    "         alpha = .5,\n",
    "         linewidth = 0, \n",
    "         label = 'Gentoo'\n",
    "        )\n",
    "plt.xlabel('t-SNE Component 1', fontsize = 14)\n",
    "plt.ylabel('t-SNE Component 2', fontsize = 14)\n",
    "plt.legend()\n",
    "#plt.grid(linestyle = '--')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualization code for the overall evaluation results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xlabels = ['Full', 'RF', 'PCA', 'LDA', 'tSNE']\n",
    "\n",
    "fig, (ax1, ax4) = plt.subplots(2, 1, figsize=(8, 4))\n",
    "fig.tight_layout()\n",
    "\n",
    "ax1.plot(f1scores, label='F1 Score', c='r')\n",
    "ax1.set(ylabel='F1 score')\n",
    "#ax1.set(xlabel='Feature selection step')\n",
    "ax1.set_xticks(np.arange(len(f1scores)), labels=xlabels)\n",
    "ax1.set_ylim([0.4, 1])\n",
    "ax1.legend()\n",
    "\n",
    "ax4.plot(roc_aucscores, label='ROC-AUC Score', c='m')\n",
    "ax4.set(ylabel='ROC-AUC score')\n",
    "#ax4.set(xlabel='Feature selection step')\n",
    "ax4.set_xticks(np.arange(len(roc_aucscores)), labels=xlabels)\n",
    "ax4.set_ylim([0.4, 1])\n",
    "ax4.legend()\n",
    "\n",
    "#fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Q6: Did feature selection/engineering help produce a dataset that is drastically easier to learn? \n",
    "- Q7: Which of PCA, LDA, t-SNE gave the weakest visual separation between the 3 classes? \n",
    "- Q8: Which of random forests, PCA, LDA, t-SNE gave the weakest dataset for classification purposes? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}