FML1w1_Iris-decision-tree-classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": false
   },
   "source": [
    "# Foundations of Machine Learning 1\n",
    "## Week 1: Introduction\n",
    "## Assignment 1: classification with the Iris dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In this first assignment of the course we will go through the basic steps of setting up and training a machine learning model. We also indicate the basic elements of running a Jupyter notebook. \n",
    "### This assignment is about classifying Iris flowers into 3 different classes depending on 4 basic characteristics that can be easily measured: sepal/petal length/width. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload this Jupyter notebook to your favourite cloud environment or to your own Python environment. We recommend using Google Collaboratory, that you case access for free at https://colab.research.google.com using your Google account. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two types of cells in the Jupyter notebooks: \"markdown\" and \"code\". The markdown cells are used for longer explanations, while the code cells contain the Python instructions that make up your code. They include importing libraries, loading the data, preparing your data, creating and training your models, estimating the final model's quality. You can run the cells one by one, or run the all at once and check their status (ran with errors or not, gave an output, etc.) individually. Comments can also be inserted into the code cells, on lines that start with \"#\" or on several lines bounded by three apostrophes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "uuid": "e2b8e959-75f0-4fa9-a878-5ab024f89223"
   },
   "outputs": [],
   "source": [
    "# Import the necessary libraries.\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Iris Dataset: https://en.wikipedia.org/wiki/Iris_flower_data_set\n",
    "\n",
    "This dataset consists of data collected on 150 Iris flowers, 50 from each of three types: Setosa, Versicolour, and Virginica. For each flower in the dataset we have its Iris type, its petal and sepal\n",
    "length and width. The goal of this assignment is to create a model that learns the type of Iris based on its petal and sepal length and width."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Import the Iris dataset from the sklearn library. \n",
    "# The organisation of the data in the sklearn library is described at https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html. \n",
    "\n",
    "from sklearn.datasets import load_iris\n",
    "iris_dataset = load_iris()\n",
    "\n",
    "#Check the \"keys\" (column names) of the dataset\n",
    "print(\"Components of the Iris dataset:\\n\", iris_dataset.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check the content of the Iris dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Check the component \"DESCR\" \n",
    "print('The full description of the dataset:\\n',iris_dataset['DESCR'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the component \"data\"\n",
    "print('The type of the feature \"data\" is',type(iris_dataset['data']))\n",
    "print('The shape of the \"data\" is',iris_dataset['data'].shape)\n",
    "print(\"Check the first five rows of data:\\n\", iris_dataset['data'][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We conclude that we have 150 data points in the dataset, each with 4 features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the component \"target\" \n",
    "print('The type of feature \"target\" is',type(iris_dataset['target']))\n",
    "print('The shape of the \"target\" is',iris_dataset['target'].shape)\n",
    "print('Check the first five targets:', iris_dataset['target'][:5])\n",
    "print('Check the distinct values of \"target\":', np.unique(iris_dataset['target']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We conclude that \"target\" is a vector with 150 entries, one for each of the corresponding rows in \"data\". There is a numerical (categorical) encoding for the type of Iris of the flower described in that data point. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the component \"target_names\" \n",
    "print('The type of feature \"target_names\" is',type(iris_dataset['target_names']))\n",
    "print('The shape of the \"target_names\" is',iris_dataset['target_names'].shape)\n",
    "print('Check the values of \"target_names\":', iris_dataset['target_names'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the component \"feature_names\" \n",
    "print('The type of feature \"feature_names\" is',type(iris_dataset['feature_names']))\n",
    "print('Check the values of \"feature_names\":', iris_dataset['feature_names'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data exploration\n",
    "#### Visualization code adapted from: https://www.kaggle.com/code/kostasmar/exploring-the-iris-data-set-scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just for visualisation purposes we will use the Iris type names, rather than their numerical encoding\n",
    "iris_dataset_target_n = iris_dataset['target'].astype(str)\n",
    "for i in range(3):\n",
    "    iris_dataset_target_n[iris_dataset_target_n == str(i)] = iris_dataset['target_names'][i]\n",
    "\n",
    "# The plotting function: a violin plot of the data, one feature at a time, grouped by the Iris type\n",
    "def plot_violin(feature, plot_location):\n",
    "    ax = plt.subplot(2,2,plot_location)    \n",
    "    ax.set(title='Data distribution:'+iris_dataset['feature_names'][feature])\n",
    "    sns.violinplot(\n",
    "        #x=iris_dataset['target'],\n",
    "        x=iris_dataset_target_n,\n",
    "        y=iris_dataset['data'][:,feature], \n",
    "    )\n",
    " \n",
    "# Plot the violin plots, for each of the 4 features\n",
    "plt.figure(figsize=(17,12))\n",
    "plot_location = 1\n",
    "for feature in range(4):\n",
    "    plot_violin(feature, plot_location)\n",
    "    plot_location += 1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We conclude, especially based on the lower 2 plots, that the 3 classes can be differentiated from each other. So machine learning has a good chance of succeeding on this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare for the machine learning phase: split the data into train/validation/test 60/20/20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reset the seed of the random number generator, for reproducibility purposes\n",
    "np.random.seed(2023)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First split the data into 20% for the testing data and 80% for the training and validation.\n",
    "# The data is split in a stratified fashion: \n",
    "#      the three classes contribute proportionally to the train and the test datasets\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train_valid, X_test, y_train_valid, y_test = train_test_split(\n",
    "    iris_dataset['data'], \n",
    "    iris_dataset['target'], \n",
    "    test_size=0.20, \n",
    "    shuffle=True,\n",
    "    random_state=100,\n",
    "    stratify=iris_dataset['target']\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the split\n",
    "\n",
    "print(\"X_train_valid shape:\", X_train_valid.shape)\n",
    "print(\"y_train_valid shape:\", y_train_valid.shape)\n",
    "print(\"X_test shape:\", X_test.shape)\n",
    "print(\"y_test shape:\", y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now split the train_valid data into  separate training and validation data.\n",
    "\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_valid, \n",
    "    y_train_valid, \n",
    "    test_size=0.25, \n",
    "    shuffle=True,\n",
    "    random_state=200,\n",
    "    stratify=y_train_valid\n",
    ")\n",
    "\n",
    "# Question: why do we set the split ratio here to 25%, \n",
    "#     when the previous split was at 20%, if we aim to get equal sized validation and test sets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the split\n",
    "\n",
    "print(\"X_train shape:\", X_train.shape)\n",
    "print(\"y_train shape:\", y_train.shape)\n",
    "print(\"X_valid shape:\", X_valid.shape)\n",
    "print(\"y_valid shape:\", y_valid.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data very often should be normalised and/or scaled to help the learning of the model. This is the right place to normalise/scale the train data, after the split is done, to avoid any data leakage. The exact same normalisation/scaling must be applied separately to the validation and to the test data.\n",
    "\n",
    "In our case, we scale each of the features to the [0, 1] range. Other choices may also work fine. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "min_max_scaler = MinMaxScaler()\n",
    "min_max_scaler = min_max_scaler.fit(X_train)\n",
    "X_train = min_max_scaler.transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train your first model: decision tree\n",
    "A decision tree is a simple machine learning model that aims to classify datapoints through a series of \"decisions\". The yes/no outcome of each decision can be seen as a binary tree, and more decisions can be taken further down on the tree. We only use on this dataset a decision tree of depth at most 2, in other words we aim to classify the Irises through two consequetive questions/decision on their feature. \n",
    "\n",
    "Decision trees will be discussed in details later in the course. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import tree\n",
    "\n",
    "clf2 = tree.DecisionTreeClassifier(\n",
    "    criterion=\"gini\", \n",
    "    max_depth=2,\n",
    "    random_state=2023\n",
    ")\n",
    "\n",
    "clf2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluating the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Test the performance on the train data\n",
    "print(\"Model score on the training data: {:.2f}\".format(clf2.score(X_train, y_train)))\n",
    "\n",
    "y_pred_train = clf2.predict(X_train)\n",
    "print(\"Predictions on the training set:\\n\", y_pred_train)\n",
    "print(\"Real labels:\\n\", y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We conclude that the model has an excellent performance on the training dataset with 98% accuracy. \n",
    "Let's check its performance on the validation dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Test the performance on the validation data\n",
    "print(\"Model score on the validation data: {:.2f}\".format(clf2.score(X_valid, y_valid)))\n",
    "\n",
    "y_pred_valid = clf2.predict(X_valid)\n",
    "print(\"Predictions on the validation set:\\n\", y_pred_valid)\n",
    "print(\"Real labels:\\n\", y_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assignment 1. \n",
    "It looks like the model has very poor performance on the validation set. Why is it so when its performance was excellent on the training set? Fix the problem so that the model shows excellent performance also on the validation set. Report the model score on the validation set in Moodle (hint: it should be over 0.95)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write here your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assignment 2\n",
    "Train decision tree models of depth 3 and 4. Check their scores on the train and on the validation datasets. Choose the most suitable model for the problem. What is its depth?\n",
    "\n",
    "You should expect that more complex models may give better scores. The question you need to consider is whether the increased complexity justifies the increase in the score. In this assignment, consider scores within 0.03 of each other as being similarly good. Choose the smallest/simplest model as your final model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write here your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assignment 3. \n",
    "What is the model score on the test set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write here your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assignment 4\n",
    "The decision tree classifier can offer a ranking of the feature importances. Intuitively, the more a feature was consulted in the decision nodes of a tree, the higher its importance. Check online in the sklearn library how to get the \"feature importances\" ranking. What are the 2 most important features of your model? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write here your code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}