FML3w5_CV.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "IP_StudentVersion_CrossValidationAssignment.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "9u-6kiWQy_Mq"
      ]
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9u-6kiWQy_Mq"
      },
      "source": [
        "# Cross-validation: Tutorial\n",
        "\n",
        "##Introduction\n",
        "This short tutorial explains how to use cross validation on a dataset to train and evaluate a model. Cross-validation is useful in cases where the number of datapoints is lacking. The method splits the data in several different ways and trains a model for each split (these splits are also known as \"folds\".) \n",
        "\n",
        "There are a number of ways to perform Cross-validation, but they all seek to remedy the reduction of datapoints by the train-validation-test splits that is vital for model evaluation. For this reason the tutorial focuses on one of these methods. A complete list of cross-validation methods available in sklearn is available on the website https://scikit-learn.org/stable/modules/cross_validation.html. The tutorial below is sampled from the material available on that page. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DMuN6jcIEclL"
      },
      "source": [
        "Learning the parameters of a prediction function and testing it on the same data is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data. This situation is called overfitting. To avoid it, it is common practice when performing a (supervised) machine learning experiment to hold out part of the available data as a test set X_test, y_test. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rp26P_nJEoSN"
      },
      "source": [
        "Let’s load the iris data set to fit a linear support vector machine on it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P4a_-ivjEux2"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn import datasets\n",
        "from sklearn import svm\n",
        "\n",
        "X, y = datasets.load_iris(return_X_y=True)\n",
        "X.shape, y.shape"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YbAP7oZbE1XI"
      },
      "source": [
        "In the traditional training strategy, we sample a training set while holding out 40% of the data for testing (evaluating) our classifier:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nrYfk-VcE3YD"
      },
      "source": [
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)\n",
        "\n",
        "print(\"The shape of the training data:\",X_train.shape, y_train.shape)\n",
        "print(\"The shape of the test data:\",X_test.shape, y_test.shape)\n",
        "\n",
        "clf = svm.SVC(kernel='linear', C=1).fit(X_train, y_train)\n",
        "\n",
        "print(\"Accuracy on the train dataset:\",clf.score(X_train,y_train))\n",
        "print(\"Accuracy on the test dataset:\",clf.score(X_test, y_test))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-eTIMMYUFGJl"
      },
      "source": [
        "##Cross-validation\n",
        "When evaluating different settings (“hyperparameters”) for estimators, such as the C setting that must be manually set for an SVM, there is still a risk of overfitting on the test set because the parameters can be tweaked until the estimator performs optimally. This way, knowledge about the test set can “leak” into the model and evaluation metrics no longer report on generalization performance. To solve this problem, yet another part of the dataset can be held out as a so-called “validation set”: training proceeds on the training set, after which evaluation is done on the validation set, and when the experiment seems to be successful, final evaluation can be done on the test set.\n",
        "\n",
        "However, by partitioning the available data into three sets, we drastically reduce the number of samples which can be used for learning the model, and the results can depend on a particular random choice for the pair of (train, validation) sets.\n",
        "\n",
        "A solution to this problem is a procedure called cross-validation (CV for short). A test set should still be held out for final evaluation, but the validation set is no longer needed when doing CV. In the basic approach, called k-fold CV, the training set is split into k smaller sets (other approaches are described below, but generally follow the same principles. The following procedure is followed for each of the k “folds”:\n",
        "\n",
        "A model is trained using k-1 of the folds as training data. The resulting model is validated on the remaining part of the data (i.e., it is used as a test set to compute a performance measure such as accuracy).\n",
        "\n",
        "The performance measure reported by k-fold cross-validation is then the average of the values computed in the loop. This approach can be computationally expensive, but does not waste too much data (as is the case when fixing an arbitrary validation set), which is a major advantage in problems such as inverse inference where the number of samples is very small."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c__QX6SqFiD1"
      },
      "source": [
        "The simplest way to use cross-validation is to call the cross_val_score  function on the estimator and the dataset.\n",
        "\n",
        "The following example demonstrates how to estimate the accuracy of a linear kernel support vector machine on the iris dataset by splitting the data, fitting a model and computing the score 5 consecutive times (with different splits each time):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QlBlJIWEFi0n"
      },
      "source": [
        "from sklearn.model_selection import cross_val_score\n",
        "\n",
        "#Create the modeel\n",
        "clf = svm.SVC(kernel='linear', C=1, random_state=42)\n",
        "\n",
        "#Run the cross-validation and collect the accuracy of each fold\n",
        "scores = cross_val_score(clf, X, y, cv=5)\n",
        "\n",
        "#Print the accuracy of each fold\n",
        "print(\"The accuracy of each fold:\",scores)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rt-q_LkCFlp_"
      },
      "source": [
        "The mean score and the standard deviation are hence given by:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YBgPjNs8Fpe-"
      },
      "source": [
        "print(\"%0.2f accuracy with a standard deviation of %0.2f\" % (scores.mean(), scores.std()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0f54OsgEKJpi"
      },
      "source": [
        "We can experiment now with a value C=10 and choose the better setup as indicated by the accuracy of the two models. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FZt9mEmNKPYI"
      },
      "source": [
        "#Create the modeel\n",
        "clf = svm.SVC(kernel='linear', C=10, random_state=42)\n",
        "\n",
        "#Run the cross-validation and collect the accuracy of each fold\n",
        "scores = cross_val_score(clf, X, y, cv=5)\n",
        "\n",
        "#Print the accuracy of each fold\n",
        "print(\"The accuracy of each fold:\",scores)\n",
        "\n",
        "print(\"%0.2f accuracy with a standard deviation of %0.2f\" % (scores.mean(), scores.std()))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KDzPbmQtFshA"
      },
      "source": [
        "##Data transformation with held out data\n",
        "\n",
        "Just as it is important to test a predictor on data held-out from training, preprocessing (such as standardization, feature selection, etc.) and similar data transformations similarly should be learnt from a training set and applied to held-out data for prediction:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Cc6rF5i1F7Os"
      },
      "source": [
        "from sklearn import preprocessing\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)\n",
        "\n",
        "#We use standardization, and we get its ooptimal setup by training on X_train \n",
        "scaler = preprocessing.StandardScaler().fit(X_train)\n",
        "\n",
        "#Apply the standardization to the train dataset\n",
        "X_train_transformed = scaler.transform(X_train)\n",
        "\n",
        "#Train the model on the standardized training dataset\n",
        "clf = svm.SVC(C=1).fit(X_train_transformed, y_train)\n",
        "print(\"Accuracy of the model on the (standardized) train data:\",clf.score(X_train_transformed, y_train))\n",
        "\n",
        "#Apply the standardization to the test dataset\n",
        "#Note: this is the scaler that we got based on the training data. \n",
        "#We must not use a different scaler on the test data or when deploying the model.\n",
        "X_test_transformed = scaler.transform(X_test)\n",
        "print(\"Accuracy of the model on the (standardized) test data:\",clf.score(X_test_transformed, y_test))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LkpQtpnbQO8K"
      },
      "source": [
        "# Cross-validation: Project\n",
        "\n",
        "In this project you will use a dataset from Kaggle (https://www.kaggle.com/fanconic/skin-cancer-malignant-vs-benign) consisting of images of skin tumours. Each image is a 224 X 244 X 3 matrix of 224x244 pixels, where the last dimension describes the color value of the different color channels (Red, Green, Blue).\n",
        "\n",
        "You will use Cross-validation to train a model in different folds to increase accuracy for malign vs. benign predictions (cancerous vs. non-cancerous) You can use any type of model you want (Supervised Clustering, Support vector Machines, Deep Neural Networks etc.)\n",
        "\n",
        "The first step is to access the data, the code below installs a library which makes this easy and downloads the data, but you will require a Kaggle username and API-key to acces this data (You may of course also download the data and upload it to colab, but the library is less tedious).\n",
        "\n",
        "1. Sign up on Kaggle to create a user there\n",
        "2. Go to your profile and select \"Account\" (in the field where Home, Competitions, Datasets etc. are located, to the far right)\n",
        "3. Scroll down to the API section and press the button \"Create New API Token\"\n",
        "\n",
        "This will download a .json file which you can open with a code edito, in it you will see a username and key. Insert these into the fields that appear in the cell below when you run it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9Kq35HaxMSBY"
      },
      "source": [
        "!pip install opendatasets\n",
        "import opendatasets as od\n",
        "od.download(\"https://www.kaggle.com/fanconic/skin-cancer-malignant-vs-benign\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EVS1a6RznIFk"
      },
      "source": [
        "The data should be available in the \"Files\"-section of your Google Colab menu, in a folder named *skin-cancer-malignant-vs-benign*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3ksbo9eOz2bg"
      },
      "source": [
        "# Libraries for loading data\n",
        "import os, cv2\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# This might be handy\n",
        "from sklearn.utils import shuffle"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "__j5stKvNHmx"
      },
      "source": [
        "#Use this SEED for all your random seed values.\n",
        "SEED=2021"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5iPBHJ3CP7Tv"
      },
      "source": [
        "# Method for loading the data\n",
        "def load_data(path = \"/content/skin-cancer-malignant-vs-benign/train\"):\n",
        "  # Necessary lists\n",
        "  labels = []\n",
        "  label_names = []\n",
        "  data = []\n",
        "\n",
        "  # Each folder in the parent folder contains examples of the relative class\n",
        "  for label, dir in enumerate(os.listdir(path)):\n",
        "\n",
        "    # So we can simply store the name of the folder as the label name\n",
        "    label_names.append(dir)\n",
        "\n",
        "    # For every file (picture) in the current directory\n",
        "    for img in os.listdir(path+\"/\" + dir):\n",
        "\n",
        "      # Save the label corresponding to the picture\n",
        "      labels.append(label)\n",
        "\n",
        "      # This reads the image using the cv2 library and converts it to RGB format\n",
        "      example = cv2.imread(path+\"/\" + dir + \"/\" + img)   # reads an image in the BGR format\n",
        "      example = cv2.cvtColor(example, cv2.COLOR_BGR2RGB)   # BGR -> RGB\n",
        "\n",
        "      # Now add the matrix to the list\n",
        "      data.append(example)\n",
        "\n",
        "  # Convert them to numpy arrays for ease of use\n",
        "  data = np.array(data)\n",
        "  labels = np.array(labels)\n",
        "\n",
        "  # Return\n",
        "  return data, labels, label_names"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "enkDbLsS95tN"
      },
      "source": [
        "# Load the training data\n",
        "X_train, y_train, label_names = load_data(\"/content/skin-cancer-malignant-vs-benign/train\")\n",
        "\n",
        "# Shuffle the data\n",
        "Your code here..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WI2XDAkvNg5_"
      },
      "source": [
        "print(\"The size of the training dataset:\", X_train.shape,y_train.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z3H17684FPXX"
      },
      "source": [
        "#The dataset is too large for a mini project. We only work on the first 500 images (from the shuffled data).\n",
        "\n",
        "Your code here...\n",
        "\n",
        "print(\"The size of the training dataset:\", X_train.shape,y_train.shape)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SedXHbURol-Q"
      },
      "source": [
        "#Print 5 pictures, just to check our dataset \n",
        "for i in range(5):\n",
        "  # We can display an example here by the imshow() method\n",
        "  plt.title(label_names[y_train[i]])\n",
        "  plt.imshow(X_train[i])\n",
        "  plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TJ89WCtbqB7a"
      },
      "source": [
        "The data requires preprocessing. The pixel values are commonly in a range between [0, 255]. Divide all values by 255 to normalize the pixel values into a range of [0, 1]. It is also better/easier to reshape the data into one vector i.e. (224 X 224) -> (50176)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XZiv6SfZqeur"
      },
      "source": [
        "#Normalize the data\n",
        "\n",
        "Your code here..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZvZkfvc1wi2D"
      },
      "source": [
        "# Reshape the data (N, 224, 224, 3) -> (N, 224^2, 3)\n",
        "\n",
        "Your code here...\n",
        "\n",
        "print(X_train.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PiwjaKxYxs6H"
      },
      "source": [
        "To further simplify the data we may want to reduce the size of each image by converting them into black and white. This will make it easier to apply the images to models such as support vector machines and random forrests. To do this you can look at each pixel value as a 3-dimensional vector and calculate the norm of it, then you can use that norm instead of the RGB-value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "taF24zd6yCj8"
      },
      "source": [
        "# Remove colors to reduce memory requirements\n",
        "print(\"Shape before color simplification:\", X_train.shape)\n",
        "X_train = np.linalg.norm(X_train, axis = 2)\n",
        "print(\"Shape after color simplification:\", X_train.shape)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9Nl_6lvY6dh_"
      },
      "source": [
        "# The images should now be gray-scaled\n",
        "#Print 5 pictures, just to check our dataset \n",
        "\n",
        "Your code here..."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hI7Tz6uBrfrS"
      },
      "source": [
        "##Task: load and pre-process the test dataset in the same wasy as we did for the training dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lyhQ7ezhAbJJ"
      },
      "source": [
        "X_test, y_test, _ = load_data(path = \"/content/skin-cancer-malignant-vs-benign/test\")\n",
        "\n",
        "# Reshape the data (N, 224, 224, 3) -> (N, 224^2, 3)\n",
        "Your code here...\n",
        "\n",
        "#Normalize the data\n",
        "Your code here...\n",
        "\n",
        "#Transform the pictures into gray-scaled \n",
        "Your code here...\n",
        "\n",
        "print(\"Shape after color simplification:\", X_test.shape,y_test.shape)\n",
        "\n",
        "# Shuffle the data\n",
        "Your code here...\n",
        "\n",
        "#Keep only 200 pictures in the test dataset (for performance reasons)\n",
        "Your code here...\n",
        "\n",
        "print(\"Shape after data reduction:\", X_test.shape,y_test.shape)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5red_tAepfv"
      },
      "source": [
        "##Task\n",
        "\n",
        "Find out which of the following settings give you the best support vector classifier on the training dataset;\n",
        "\n",
        "*   Kernel: rbf, linear, polynomial with degree 2, 3, or 4\n",
        "*   C hyper-parameter: 0.1, 1, or 10\n",
        "*   Gamma hyper-parameter (for rbf and poly): 'scale', 'auto', or 0.01\n",
        "\n",
        "To compare between these different models, run a 5-fold cross-validation on the training dataset and compare the accuracy of each of the models. \n",
        "\n",
        "\n",
        "\n",
        "*   Q1: What is the kernel in the best model?\n",
        "*   Q2: What is the C hyper-parameter in the best model?\n",
        "*   Q3: What is the gamma hyper-parameter in the best model?\n",
        "*   Q4: What is the accuracy of the best model on the test dataset?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8uuaCNmuh5ju"
      },
      "source": [
        "Your code here..."
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}