02_Data_Preparation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Lab 2: Data, data, data!\n",
        "\n",
        "As we discussed in our lecture, data can make or break a Machine Learning project. In this lab, we will walk through some fundamental steps that must be taken (or at least checked if they need to be done) before we begin model training.\n",
        "\n",
        "## 1. Data Cleaning\n",
        "We first need to address issues such as missing values, inconsistencies, and outliers. This ensures that the data is clean, consistent, and ready for analysis or modeling.\n",
        "\n",
        "Throughout this tutorial, we'll be using a dataset about fantasy world creatures. This dataset includes fictional creatures, their magical power levels, habitats, and more. Our goal is to prepare this dataset to answer the question, **can we predict if they have magic?** First, we will perform some exploratory data analysis (EDA) to understand the data better and visualize interesting aspects.\n",
        "\n",
        "The dataset contains the following columns:\n",
        "- `Creature`: Name of the creature.\n",
        "- `Power_Level`: Magical power level of the creature (scale from 1 to 100).\n",
        "- `Age`: Age of the creature in years.\n",
        "- `Wingspan`: Wingspan of the creature in meters.\n",
        "- `Has_Magic`: Whether the creature has magical abilities (1 for yes, 0 for no).\n",
        "\n",
        "```python\n",
        "# Import libraries we will need\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from scipy import stats\n",
        "\n",
        "# Create a DataFrame with synthetic data\n",
        "data = {\n",
        "    'Creature': ['Dragon', 'Dragon', 'Dragon', 'Unicorn', 'Unicorn', 'Phoenix', 'Phoenix', 'Goblin', 'Goblin', 'Elf',\n",
        "                 'Elf', 'Troll', 'Troll', 'Griffin', 'Griffin', 'Hobbit', 'Hobbit', 'Giant', 'Giant', 'Sphinx', 'Sphinx'],\n",
        "    'Power_Level': [95, 93, 98, 85, 87, 90, None, 50, 55, 70, 72, 40, 45, 60, 65, 55, None, 80, None, 75, 80],\n",
        "    'Age': [300, 310, 290, 5000, 155, 100, None, 50, 55, 200, 210, 75, 80, 120, 125, None, 2000, 500, None, 2000],\n",
        "    'Wingspan': [12.5, 13.0, None, None, None, 15.0, 16.0, 2.0, 50.0, None, 2.2, None, 3.0, 7.0, 8.0, 1.5, 1.8, None, 10.0, 11.0],\n",
        "    'Has_Magic': [1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1]\n",
        "}\n",
        "\n",
        "# create a DataFrame\n",
        "df = pd.DataFrame(data)\n",
        "\n",
        "# view the first 5 rows\n",
        "df.head() # or df.tail() to view the last 5 rows\n",
        "```\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Create a new code cell and create this dataset for yourself to use (don't forget to import the pandas package and make the data into a DataFrame)."
      ],
      "metadata": {
        "id": "NI-O0O47rsg2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.1. Exploratory Data Analysis\n",
        "\n",
        "#### Basic Info\n",
        "\n",
        "Let's see some general information about this data. Here are some easy was to get a sense of what you are working with:\n",
        "\n",
        "```python\n",
        "# Get basic information about the dataset\n",
        "df.info()\n",
        "\n",
        "# Summary statistics of the dataset\n",
        "df.describe(include='all')\n",
        "```\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Use these functions to understand some basics about the data."
      ],
      "metadata": {
        "id": "SNxiz5YJgY6H"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Getting number of observations per value\n",
        "We can see how many observations (rows) have each value within a particular varibale using the `value_counts()` function.\n",
        "\n",
        "```python\n",
        "value_counts = df['column_name'].value_counts()\n",
        "print(value_counts)\n",
        "```\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Explore each variable. Do you notice any large repeats?"
      ],
      "metadata": {
        "id": "l92uT4TvsL47"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Distribution of a variable\n",
        "Here is a simple way to visualize the distribution of a variable, AAA.\n",
        "\n",
        "```python\n",
        "# Create a histogram for the Power Level distribution\n",
        "plt.figure(figsize=(8, 6))\n",
        "plt.hist(df['AAA'].dropna(), bins=10, color='purple', edgecolor='black')\n",
        "\n",
        "# add some aesthetics\n",
        "plt.title('Distribution of AAA')\n",
        "plt.xlabel('AAA')\n",
        "plt.ylabel('Frequency')\n",
        "\n",
        "# show the plot\n",
        "plt.show()\n",
        "```\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Pick a variable (or try it for a few) that has is a `float` and plot it's distribution. Is it normal?"
      ],
      "metadata": {
        "id": "t2ckdIHMg7e2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Using scatter plots\n",
        "Let's see if there is a relationship between three variables (`AAA`, `BBB`, `CCC`) by visualizing the data as a scatterplot.\n",
        "\n",
        "```python\n",
        "# set plot configurations\n",
        "plt.figure(figsize=(10, 6))\n",
        "colors = {0: 'red', 1: 'green'}\n",
        "\n",
        "# plot the data\n",
        "plt.scatter(df['AAA'], df['BBB'], c=df['CCC'].map(colors), alpha=0.7)\n",
        "\n",
        "# add aesthetics\n",
        "plt.title('AAA vs. BBB')\n",
        "plt.xlabel('AAA')\n",
        "plt.ylabel('BBB')\n",
        "plt.legend(handles=[plt.Line2D([0], [0], marker='o', color='w', markerfacecolor='blue', markersize=10, label='CCC-0'),\n",
        "                    plt.Line2D([0], [0], marker='o', color='w', markerfacecolor='orange', markersize=10, label='CCC-1')],\n",
        "           loc='best')\n",
        "plt.grid(True)\n",
        "\n",
        "# show the image\n",
        "plt.show()\n",
        "```\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Pick three variables and test this method out. Remember that the variable you choose for CCC must be categorical!"
      ],
      "metadata": {
        "id": "vSk-L05plhQP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Using bar plots\n",
        "Calculate average of a numerical variable for each value of a categorical variable.\n",
        "\n",
        "```python\n",
        "# group by BBB (categorical) and get the mean AAA (continuous) for each\n",
        "avg_AAA_by_BBB = df.groupby('BBB')['AAA'].mean()\n",
        "\n",
        "# create a bar plot\n",
        "plt.figure(figsize=(10, 6))\n",
        "avg_AAA_by_BBB.plot(kind='bar')\n",
        "\n",
        "# add aesthetics\n",
        "plt.title('Average AAA by BBB')\n",
        "plt.xlabel('BBB')\n",
        "plt.ylabel('AAA')\n",
        "\n",
        "# show the plot\n",
        "plt.show()\n",
        "```\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Are there two particular variables that you think this would be useful for? See if there are any trends."
      ],
      "metadata": {
        "id": "oScjW0YDmczF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.2. Handling Missing Values\n",
        "\n",
        "Now that we've explored our data a bit, we can begin to make changes to it. We'll start with missing values. There are two options: Drop or Impute.\n",
        "\n",
        "**Drop Missing Values**\n",
        "\n",
        "One option is to remove rows or columns that contain missing values. This is useful when the amount of missing data is small.\n",
        "\n",
        "```python\n",
        "# Drop rows with missing values\n",
        "df.dropna(inplace=True)\n",
        "```\n",
        "**Impute Missing Values**\n",
        "\n",
        "Filling in missing values with the mean (`.mean()`), median (`.mode()`), or mode (`.mode()`) is a common approach. More sophisticated imputation methods (e.g., kNN, MICE) can also be used but we will keep things simple here. This is helpful if you have less data in general, or if removing missing values would cause imbalances.\n",
        "\n",
        "```python\n",
        "# Calculate the mean of a column\n",
        "column_mean = df['AAA'].mean()\n",
        "\n",
        "# Fill missing values in the column with the mean\n",
        "df['AAA'].fillna(column_mean, inplace=True)\n",
        "```\n",
        "Note:\n",
        "- `inplace=True`: This modifies the original DataFrame df. If you don’t want to modify the original DataFrame, you can set `inplace=False` (or omit the parameter) and assign the result to a new DataFrame.\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Take care of all the missing values in this dataset. Be sure that you can justify your choices!"
      ],
      "metadata": {
        "id": "yACVukRUgdk9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.3. Removing Duplicates\n",
        "Check for and remove duplicate rows to ensure that each entry in your dataset is unique. Duplicates cause biases in models.\n",
        "\n",
        "```python\n",
        "# Remove duplicate rows\n",
        "df.drop_duplicates(inplace=True)\n",
        "```\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Drop duplicates and check how many were removed!"
      ],
      "metadata": {
        "id": "ezrc5AZKggtW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.4. Correcting Data Types\n",
        "Ensure each column has the appropriate data type. This can be critical for accurate analysis and modeling.\n",
        "\n",
        "```python\n",
        "# Convert data types to int, float, and categorical\n",
        "df['AAA'] = df['AAA'].astype(int)\n",
        "df['AAA'] = df['AAA'].astype(float)\n",
        "df['AAA'] = df['AAA'].astype('category')\n",
        "```\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Make sure all data types are correct (Hint: There was a method above that showed you the data type of each variable). If they are't correct, go ahead and fix them."
      ],
      "metadata": {
        "id": "1FHaflX1gjjq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.5. Handling Outliers\n",
        "Outliers can skew the data distribution and affect model performance. Common methods to identify and handle outliers include using Z-scores and the Interquartile Range (IQR).\n",
        "\n",
        "**Z-Score Method**\n",
        "\n",
        "```python\n",
        "from scipy import stats\n",
        "\n",
        "# Calculate Z-scores\n",
        "df['z_score'] = stats.zscore(df['AAA'])\n",
        "\n",
        "# Define a threshold for Z-scores (3 is standard)\n",
        "threshold = 3\n",
        "\n",
        "# Identify outliers\n",
        "df['is_outlier'] = np.abs(df['z_score']) > threshold\n",
        "\n",
        "# Drop outliers\n",
        "df_cleaned = df[~df['is_outlier']].drop(columns=['z_score', 'is_outlier'])\n",
        "```\n",
        "**IQR Method**\n",
        "\n",
        "```python\n",
        "# Calculate quartiles\n",
        "Q1 = df['AAA'].quantile(0.25)\n",
        "Q3 = df['AAA'].quantile(0.75)\n",
        "\n",
        "# Calculate IQR\n",
        "IQR = Q3 - Q1\n",
        "\n",
        "# Define the outlier thresholds\n",
        "lower_bound = Q1 - 1.5 * IQR\n",
        "upper_bound = Q3 + 1.5 * IQR\n",
        "\n",
        "# Identify outliers\n",
        "df['is_outlier'] = (df['values'] < lower_bound) | (df['values'] > upper_bound)\n",
        "df.head()\n",
        "\n",
        "# Drop outliers\n",
        "df_cleaned = df[(df['values'] >= lower_bound) & (df['values'] <= upper_bound)]\n",
        "```\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Identify and handle outliers. Be sure that you can justify why you used the methods you did!"
      ],
      "metadata": {
        "id": "5JqMc2xbgnEF"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1.6. Encoding Categorical Variables\n",
        "Convert categorical data into numerical format using techniques like one-hot encoding.\n",
        "\n",
        "```python\n",
        "# Perform one-hot encoding\n",
        "df_encoded = pd.get_dummies(df, columns=['AAA', 'BBB'])\n",
        "print(df_encoded)\n",
        "```\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Encode the `Creature` variable."
      ],
      "metadata": {
        "id": "zHl6_XhrgqiE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Feature Engineering for Machine Learning\n",
        "Feature engineering is a crucial step in the machine learning pipeline that transforms raw data into meaningful features, significantly impacting model performance. This lesson covers key aspects of feature engineering, including feature extraction, scaling, normalization, and selection techniques."
      ],
      "metadata": {
        "id": "liAj94COxPUm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.1. Feature Extraction\n",
        "Feature extraction involves transforming raw data into a set of features that can be used for machine learning models. This process helps in capturing relevant information and can lead to improved model performance.\n",
        "\n",
        "#### Common Feature Extraction Techniques\n",
        "\n",
        "***Note***: The below techniques are used for both classification and regression tasks.\n",
        "\n",
        "**Principal Component Analysis (PCA)** is a dimensionality reduction technique that transforms data into a set of linearly uncorrelated components, maximizing the variance. Choosing 2 for the number of components is a common practice when you want to visualize high-dimensional data in a 2D plot. It allows you to plot the data points on a 2D plane, which helps in understanding the structure and distribution of the data. Reducing to 2 dimensions is often used for exploratory data analysis and visualization.\n",
        "\n",
        "```python\n",
        "from sklearn.decomposition import PCA\n",
        "\n",
        "# Initialize PCA\n",
        "pca = PCA(n_components=2)  # Reducing to 2 dimensions\n",
        "df_pca = pca.fit_transform(df)\n",
        "\n",
        "# see how much variance is explained\n",
        "print(\"Explained variance ratio:\", pca.explained_variance_ratio_)\n",
        "\n",
        "# inspect the  components to understand which original features are most influential in the new component space\n",
        "print(\"Principal components:\\n\", pca.components_)\n",
        "```\n",
        "This will result in an entirely new set of features. Since we are early on in the class, and we want to keep some interpretability to our features, we will skip this for right now. But remember that this code is here when you need it in the future!"
      ],
      "metadata": {
        "id": "lc8LJmMK4dJ4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Polynomial Features** creates new features by considering polynomial combinations of existing features, capturing interactions between them.\n",
        "\n",
        "```python\n",
        "from sklearn.preprocessing import PolynomialFeatures\n",
        "\n",
        "# Columns to apply polynomial features\n",
        "columns_to_transform = ['AAA', 'BBB']\n",
        "\n",
        "# Separate the features\n",
        "features_to_transform = df[columns_to_transform]\n",
        "features_not_to_transform = df.drop(columns=columns_to_transform)\n",
        "\n",
        "# Initialize PolynomialFeatures\n",
        "poly = PolynomialFeatures(degree=2)  # add quadratic features\n",
        "\n",
        "# Fit and transform the specified features\n",
        "poly_features = poly.fit_transform(features_to_transform)\n",
        "\n",
        "# Create DataFrame for the polynomial features\n",
        "poly_features_df = pd.DataFrame(poly_features, columns=poly.get_feature_names_out(columns_to_transform), index=df.index)\n",
        "\n",
        "# Combine polynomial features with non-transformed features\n",
        "df_transformed = pd.concat([poly_features_df, features_not_to_transform], axis=1)\n",
        "```\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Create one or more polynomial features from our current `df`. Be sure that you can justify your choice."
      ],
      "metadata": {
        "id": "02gcvraS4g--"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Feature Engineering from Dates**\n",
        "Extract meaningful components from date-time variables such as year, month, day, and weekday.\n",
        "\n",
        "```python\n",
        "# Sample data (note: there are other date-time formats, but this is most common)\n",
        "datedf = pd.DataFrame({'date': pd.to_datetime(['2023-01-01', '2023-02-01', '2023-03-01'])})\n",
        "\n",
        "# Extract features\n",
        "datedf['year'] = datedf['date'].dt.year\n",
        "datedf['month'] = datedf['date'].dt.month\n",
        "datedf['day'] = datedf['date'].dt.day\n",
        "datedf['weekday'] = datedf['date'].dt.weekday\n",
        "\n",
        "print(datedf.head())\n",
        "```\n",
        "\n",
        "We don't have a date variable in our current `df`, but keep this in mind for Assignment #1!"
      ],
      "metadata": {
        "id": "3kA1xvq24kmQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.2. Feature Scaling\n",
        "Feature scaling ensures all features contribute equally to model training. It helps improve the performance of algorithms sensitive to feature scales. Both of the below techniques are used in both classification and regression tasks.\n",
        "\n",
        "#### Key Techniques\n",
        "\n",
        "**Standardization (Z-score Normalization)**\n",
        "\n",
        "Transforms features to have a mean of 0 and a standard deviation of 1.\n",
        "\n",
        "```python\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "# columns to scale\n",
        "columns_to_scale = ['AAA', 'BBB']\n",
        "\n",
        "# separate the features\n",
        "features_to_scale = df[columns_to_scale]\n",
        "features_not_to_scale = df.drop(columns=columns_to_scale)\n",
        "\n",
        "# initialize and apply StandardScaler\n",
        "scaler = StandardScaler()\n",
        "scaled_features = scaler.fit_transform(features_to_scale)\n",
        "\n",
        "# convert scaled features back to DataFrame\n",
        "scaled_features_df = pd.DataFrame(scaled_features, columns=columns_to_scale, index=df.index)\n",
        "\n",
        "# Combine scaled and non-scaled features\n",
        "df_scaled = pd.concat([scaled_features_df, features_not_to_scale], axis=1)\n",
        "\n",
        "print(df_scaled)\n",
        "```"
      ],
      "metadata": {
        "id": "YVJyKRYB4oCQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Min-Max Scaling (Normalization)**\n",
        "\n",
        "Scales features to a fixed range, usually [0, 1].\n",
        "\n",
        "*Same process as above with slight changes:*\n",
        "\n",
        "```python\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "\n",
        "# pick and separate the features here\n",
        "\n",
        "# Initialize and apply MinMaxScaler\n",
        "scaler = MinMaxScaler()\n",
        "normalized_features = scaler.fit_transform(features)\n",
        "\n",
        "# continue by adding the normalized features back into the dataset\n",
        "```\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Compute both Standardization and Min-Max Scaling on a subset of features in the data."
      ],
      "metadata": {
        "id": "JLsg6VE94rTq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**When to Use Different Scaling Techniques**\n",
        "\n",
        "***Standardization***: Use when features have different units or scales, and you need to normalize for algorithms sensitive to feature magnitudes.\n",
        "\n",
        "***Min-Max Scaling***: Use when features need to be bounded within a specific range, especially for algorithms assuming features are within a fixed range."
      ],
      "metadata": {
        "id": "hvLhO1lo4t8c"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.3. Feature Normalization\n",
        "Normalization adjusts features to fit a specific range or achieve a certain norm, often useful for algorithms relying on distance metrics.\n",
        "\n",
        "#### Techniques\n",
        "\n",
        "**L2 Normalization (Vector Normalization)**\n",
        "Scales features so that the sum of squares of feature values (across the rows) is 1. In other words, this has to be done across all features.\n",
        "\n",
        "```python\n",
        "from sklearn.preprocessing import Normalizer\n",
        "\n",
        "columns_to_normalize = ['AAA', 'BBB']\n",
        "features_to_normalize = df[columns_to_normalize]\n",
        "\n",
        "# Initialize and apply Normalizer\n",
        "normalizer = Normalizer(norm='l2')\n",
        "normalized_features = normalizer.fit_transform(features_to_normalize)\n",
        "\n",
        "# convert scaled features back to DataFrame\n",
        "scaled_features_df = pd.DataFrame(normalized_features, columns=columns_to_normalize, index=df.index)\n",
        "\n",
        "print(scaled_features_df)\n",
        "\n",
        "# add those normalized features back to the dataset\n",
        "```"
      ],
      "metadata": {
        "id": "HSZq498V44K5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**L1 Normalization**\n",
        "\n",
        "Scales features so that the sum of absolute values of the features is 1. Same code process as above except change the norm:\n",
        "\n",
        "```python\n",
        "# Initialize and apply Normalizer\n",
        "normalizer = Normalizer(norm='l1')\n",
        "```\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Which method is most appropriate? Apply it!"
      ],
      "metadata": {
        "id": "ePg68rbd4xrw"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Summary of Differences**\n",
        "\n",
        "***Feature Scaling***: Adjusts the range or distribution of feature values. Techniques include standardization, min-max scaling, robust scaling, and maxabs scaling.\n",
        "\n",
        "***Feature Normalization***: Typically adjusts features to fit a specific range or achieve a certain norm. Techniques include min-max normalization, L1 normalization, and L2 normalization."
      ],
      "metadata": {
        "id": "Cpo6DpkEH8cL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2.4. Feature Selection\n",
        "Feature selection involves choosing the most relevant features to improve model performance, reduce overfitting, and decrease training time."
      ],
      "metadata": {
        "id": "IvdKdy07H_8D"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Filter Methods\n",
        "Evaluate the relevance of features based on statistical tests.\n",
        "\n",
        "**Chi-Square Test** is used for classification tasks and assesses associations between *categorical variables*.\n",
        "\n",
        "```python\n",
        "from sklearn.feature_selection import SelectKBest, chi2\n",
        "\n",
        "# Sample data\n",
        "X = pd.DataFrame({'feature1': [tall, short, short, short], 'feature2': [blue, green, red, blue]}, {'feature3': [.....)\n",
        "y = [0, 1, 0, 1]\n",
        "\n",
        "# Initialize and apply SelectKBest\n",
        "selector = SelectKBest(score_func=chi2, k=1)  # Select top 1 feature\n",
        "X_new = selector.fit_transform(X, y)\n",
        "\n",
        "print(\"Selected feature indices:\", selector.get_support(indices=True))\n",
        "```\n",
        "\n",
        "**ANOVA F-test** is used to evaluate the relevance of each feature. Features with high F-values are considered more relevant for predicting the target variable. This is used for *classification* and most commonly for *continuous features* (it can be used for categorical if they are encoded).\n",
        "\n",
        "```python\n",
        "from sklearn.feature_selection import f_classif\n",
        "\n",
        "# Apply ANOVA F-value (for a singluar feature)\n",
        "feature = df[['AAA']]\n",
        "y = df['YYY']\n",
        "\n",
        "F_value, p_value = f_classif(feature, y)\n",
        "print(F_value, p_value)\n",
        "\n",
        "# To apply to all features, must split apart the features and outcome variable into own dfs\n",
        "F_values, p_values = f_classif(features_dataframe, y)\n",
        "```\n",
        "\n",
        "**Correlation Coefficient** is a method used to identify which features have the strongest linear relationships with the target variable. This is used for *regression* and *continuous features*.\n",
        "\n",
        "```python\n",
        "# Compute the correlation matrix\n",
        "correlation_matrix = df.corr()\n",
        "\n",
        "# Get correlation of each feature with the target variable\n",
        "target_correlation = correlation_matrix['YYY'].drop('YYY')  # Drop the target itself\n",
        "\n",
        "# Print correlation coefficients\n",
        "print(\"Correlation coefficients with the target variable:\")\n",
        "print(target_correlation)\n",
        "\n",
        "# Select features with high correlation coefficients\n",
        "# For example, selecting features with correlation coefficient greater than 0.5 or less than -0.5\n",
        "selected_features = target_correlation[abs(target_correlation) > 0.5].index\n",
        "```\n",
        "\n",
        "#### <font color='red'>**TRY IT**</font> &#x1f9e0;: Which method of these is most appropriate? Go ahead and apply it."
      ],
      "metadata": {
        "id": "eoEOrZUDMufU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Wrapper Methods\n",
        "Evaluate feature subsets by training a model and assessing performance. We will not be using this so much (as it requires more computational resources), but just so that you have some example code if you decide to explore it:\n",
        "\n",
        "**Recursive Feature Elimination (RFE)**\n",
        "\n",
        "```python\n",
        "from sklearn.feature_selection import RFE\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "# sample data\n",
        "X = pd.DataFrame({'feature1': [1, 2, 3, 4], 'feature2': [5, 6, 7, 8]})\n",
        "y = [0, 1, 0, 1]\n",
        "\n",
        "# initialize model and RFE\n",
        "model = LogisticRegression()\n",
        "selector = RFE(model, n_features_to_select=1)\n",
        "X_new = selector.fit_transform(X, y)\n",
        "\n",
        "print(\"Selected features:\", selector.support_)\n",
        "```"
      ],
      "metadata": {
        "id": "3kpR06-_IE39"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Embedded Methods\n",
        "\n",
        "Perform feature selection as part of the model training process. We will not go over these in depth, as we need to understand some more about regression and classification first. But here is some example code to familiarize yourself with.\n",
        "\n",
        "**Lasso Regression (L1 Regularization)**\n",
        "\n",
        "Regression Example: Predict house prices using the California housing dataset.\n",
        "\n",
        "```python\n",
        "from sklearn.datasets import fetch_california_housing\n",
        "from sklearn.linear_model import Lasso\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "# Load the California Housing dataset\n",
        "data = fetch_california_housing()\n",
        "X = pd.DataFrame(data.data, columns=data.feature_names)\n",
        "y = pd.Series(data.target, name='Price')\n",
        "\n",
        "# Standardize the features\n",
        "scaler = StandardScaler()\n",
        "X_scaled = scaler.fit_transform(X)\n",
        "\n",
        "# Split data into training and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.3, random_state=42)\n",
        "\n",
        "# Apply Lasso Regression\n",
        "lasso = Lasso(alpha=0.1)  # alpha is the regularization parameter\n",
        "lasso.fit(X_train, y_train)\n",
        "\n",
        "# Print the coefficients\n",
        "print(\"Lasso coefficients:\")\n",
        "print(lasso.coef_)\n",
        "\n",
        "# Evaluate the model\n",
        "y_pred = lasso.predict(X_test)\n",
        "print(\"Mean Squared Error:\", mean_squared_error(y_test, y_pred))\n",
        "\n",
        "# Feature selection\n",
        "selected_features = X.columns[lasso.coef_ != 0]\n",
        "print(\"Selected features:\", selected_features)\n",
        "```\n",
        "\n",
        "Classification Example: Predict species of Iris flowers using the Iris dataset.\n",
        "\n",
        "```python\n",
        "from sklearn.datasets import load_iris\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "# Load the dataset\n",
        "data = load_iris()\n",
        "X = pd.DataFrame(data.data, columns=data.feature_names)\n",
        "y = pd.Series(data.target, name='species')\n",
        "\n",
        "# Split data into training and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",
        "\n",
        "# Apply Logistic Regression with L1 Regularization\n",
        "lasso = LogisticRegression(penalty='l1', solver='liblinear')  # liblinear solver supports L1 penalty\n",
        "lasso.fit(X_train, y_train)\n",
        "\n",
        "# Print the coefficients\n",
        "print(\"Lasso coefficients:\")\n",
        "print(lasso.coef_)\n",
        "\n",
        "# Evaluate the model\n",
        "y_pred = lasso.predict(X_test)\n",
        "print(\"Accuracy:\", accuracy_score(y_test, y_pred))\n",
        "\n",
        "# Feature selection\n",
        "selected_features = X.columns[(lasso.coef_ != 0).any(axis=0)]\n",
        "print(\"Selected features:\", selected_features)\n",
        "```\n",
        "\n",
        "**Random Forest**\n",
        "\n",
        "Regression Example: Predict house prices using the Boston housing dataset.\n",
        "\n",
        "```python\n",
        "import pandas as pd\n",
        "from sklearn.ensemble import RandomForestRegressor\n",
        "from sklearn.datasets import fetch_california_housing\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "# Load the California Housing dataset\n",
        "data = fetch_california_housing()\n",
        "X = pd.DataFrame(data.data, columns=data.feature_names)\n",
        "y = pd.Series(data.target, name='Price')\n",
        "\n",
        "# Standardize the features\n",
        "scaler = StandardScaler()\n",
        "X_scaled = scaler.fit_transform(X)\n",
        "\n",
        "# Split data into training and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, test_size=0.3, random_state=42)\n",
        "\n",
        "# Apply Random Forest Regressor\n",
        "rf = RandomForestRegressor(n_estimators=100, random_state=42)\n",
        "rf.fit(X_train, y_train)\n",
        "\n",
        "# Print feature importances\n",
        "print(\"Feature importances:\")\n",
        "importances = rf.feature_importances_\n",
        "for feature, importance in zip(X.columns, importances):\n",
        "    print(f\"{feature}: {importance:.4f}\")\n",
        "\n",
        "# Evaluate the model\n",
        "y_pred = rf.predict(X_test)\n",
        "print(\"Mean Squared Error:\", mean_squared_error(y_test, y_pred))\n",
        "```\n",
        "\n",
        "Classification Example: Predict species of Iris flowers using the Iris dataset.\n",
        "\n",
        "```python\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.datasets import load_iris\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score\n",
        "\n",
        "# Load the dataset\n",
        "data = load_iris()\n",
        "X = pd.DataFrame(data.data, columns=data.feature_names)\n",
        "y = pd.Series(data.target, name='species')\n",
        "\n",
        "# Split data into training and test sets\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)\n",
        "\n",
        "# Apply Random Forest Classifier\n",
        "rf = RandomForestClassifier(n_estimators=100, random_state=42)\n",
        "rf.fit(X_train, y_train)\n",
        "\n",
        "# Print feature importances\n",
        "print(\"Feature importances:\")\n",
        "importances = rf.feature_importances_\n",
        "for feature, importance in zip(X.columns, importances):\n",
        "    print(f\"{feature}: {importance}\")\n",
        "\n",
        "# Evaluate the model\n",
        "y_pred = rf.predict(X_test)\n",
        "print(\"Accuracy:\", accuracy_score(y_test, y_pred))\n",
        "```"
      ],
      "metadata": {
        "id": "rj8bv8daIG2h"
      }
    }
  ]
}